2024-09-15T03:29:32.471 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T03:29:32.493 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:29:32.588 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: skanta@redhat.com first_in_suite: false job_id: '7905296' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - mons down - mon down - out of quorum sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNS5IfczFsQkpWcXWMM/nl3nQs9yaDUw9Fs/4qAacZ0wie8NcAueKBCPXPsUDrJXuI4BhhD26HnHonWJ17nqfoI= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKW3rFnRTPQaoJAJ4HiOij90a231Rz749ywM96UK8r9PlhAt1ERSkD0lrlOa16C5/TPNF7Fu6zLPkbhRs/YJ94= smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBEjjg5rB/1inX87OniseCpFYKNd1SlbYYLJYN2JyrHK6uN7lmkXNKSCc7IpXymQoJXY/BMo7cq6WQeXCWMoJE= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS6yCoZepRmSPiKoN/amG2AnBigz51knwJTC7ZDatklNcKJpnfKUhTlzMhr5UWG7TijdNsDR/FHL6RDtN+Ogic= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T03:29:32.588 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T03:29:32.589 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T03:29:32.589 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T03:29:32.590 INFO:teuthology.task.internal:Checking packages... 2024-09-15T03:29:32.613 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T03:29:32.614 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T03:29:32.614 INFO:teuthology.packaging:ref: None 2024-09-15T03:29:32.614 INFO:teuthology.packaging:tag: None 2024-09-15T03:29:32.614 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:29:32.614 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:29:32.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:29:32.765 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T03:29:32.766 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T03:29:32.772 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T03:29:32.773 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T03:29:32.779 INFO:teuthology.task.internal:Saving configuration 2024-09-15T03:29:32.796 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T03:29:32.802 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T03:29:32.824 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:24:33.738991', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNS5IfczFsQkpWcXWMM/nl3nQs9yaDUw9Fs/4qAacZ0wie8NcAueKBCPXPsUDrJXuI4BhhD26HnHonWJ17nqfoI='} 2024-09-15T03:29:32.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:24:33.736544', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKW3rFnRTPQaoJAJ4HiOij90a231Rz749ywM96UK8r9PlhAt1ERSkD0lrlOa16C5/TPNF7Fu6zLPkbhRs/YJ94='} 2024-09-15T03:29:32.871 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:24:33.737778', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBEjjg5rB/1inX87OniseCpFYKNd1SlbYYLJYN2JyrHK6uN7lmkXNKSCc7IpXymQoJXY/BMo7cq6WQeXCWMoJE='} 2024-09-15T03:29:32.891 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:24:33.734919', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS6yCoZepRmSPiKoN/amG2AnBigz51knwJTC7ZDatklNcKJpnfKUhTlzMhr5UWG7TijdNsDR/FHL6RDtN+Ogic='} 2024-09-15T03:29:32.891 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T03:29:32.898 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-15T03:29:32.899 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-15T03:29:32.899 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-15T03:29:32.899 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-15T03:29:32.899 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T03:29:33.019 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbac5783ac0>, signals=[15]) 2024-09-15T03:29:33.019 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T03:29:33.027 INFO:teuthology.task.internal:Opening connections... 2024-09-15T03:29:33.027 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:29:33.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:29:33.124 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:29:33.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:29:33.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:29:33.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:29:33.280 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:29:33.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:29:33.359 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T03:29:33.365 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-09-15T03:29:33.384 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-09-15T03:29:33.384 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-09-15T03:29:33.443 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-09-15T03:29:33.443 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="9" 2024-09-15T03:29:33.443 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-09-15T03:29:33.443 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:29:33.443 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="9" 2024-09-15T03:29:33.443 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:29:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:29:33.445 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-09-15T03:29:33.467 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-09-15T03:29:33.487 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-09-15T03:29:33.487 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-09-15T03:29:33.544 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:29:33.545 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:29:33.546 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:29:33.546 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-09-15T03:29:33.567 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-09-15T03:29:33.585 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-09-15T03:29:33.586 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-09-15T03:29:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-09-15T03:29:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-09-15T03:29:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-09-15T03:29:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:29:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:29:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:29:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:29:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:29:33.646 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-09-15T03:29:33.667 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-09-15T03:29:33.685 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-09-15T03:29:33.686 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:29:33.744 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-09-15T03:29:33.745 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:29:33.745 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:29:33.745 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:29:33.745 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:29:33.745 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:29:33.745 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:29:33.745 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-09-15T03:29:33.767 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T03:29:33.776 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T03:29:33.800 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T03:29:33.800 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:29:33.802 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:29:33.805 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:29:33.807 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:29:33.824 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T03:29:33.830 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T03:29:33.830 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:29:33.862 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:29:33.864 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:29:33.867 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:29:33.890 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T03:29:33.985 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T03:29:33.999 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T03:29:33.999 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T03:29:34.000 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:29:34.000 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:29:34.001 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:29:34.001 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:29:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:29:34.018 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-15T03:29:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:29:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:29:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:29:34.020 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-09-15T03:29:34.020 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-15T03:29:34.021 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-15T03:29:34.076 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:29:34.076 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.14.0-503.el9.x86_64 2024-09-15T03:29:34.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-09-15T03:29:34.077 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:29:34.078 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-503.el9.x86_64 2024-09-15T03:29:34.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-09-15T03:29:34.079 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:29:34.079 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:29:34.080 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-503.el9.x86_64 2024-09-15T03:29:34.080 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-09-15T03:29:34.080 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-503.el9.x86_64 2024-09-15T03:29:34.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-09-15T03:29:36.705 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.2 MB 00:00 2024-09-15T03:29:37.492 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.2 MB 00:01 2024-09-15T03:29:37.544 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-15T03:29:37.860 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.2 MB 00:02 2024-09-15T03:29:39.420 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T03:29:39.484 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-15T03:29:41.247 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 20 MB 00:03 2024-09-15T03:29:44.202 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.5 MB 00:00 2024-09-15T03:29:44.436 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 20 MB 00:05 2024-09-15T03:29:45.669 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-15T03:29:46.346 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.5 MB 00:03 2024-09-15T03:29:46.581 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-15T03:29:46.840 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-15T03:29:47.830 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-15T03:29:48.340 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-15T03:29:48.868 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-15T03:29:49.236 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T03:29:50.137 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-15T03:29:51.821 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-15T03:29:52.146 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-15T03:29:52.691 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-15T03:29:54.261 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:29:54.261 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:29:54.282 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-15T03:29:54.347 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:29:54.356 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 79 M 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 126 M 2024-09-15T03:29:54.357 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:29:54.697 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 254 kB/s | 23 kB 00:00 2024-09-15T03:29:54.809 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:29:56.288 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:29:56.288 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:29:56.374 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:29:56.384 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:29:56.384 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:29:56.385 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:29:56.386 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-09-15T03:29:56.386 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:29:56.386 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 79 M 2024-09-15T03:29:56.386 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 126 M 2024-09-15T03:29:56.386 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:29:56.742 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 228 kB/s | 23 kB 00:00 2024-09-15T03:29:56.940 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:29:56.940 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:29:57.040 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:29:57.052 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:29:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-09-15T03:29:57.052 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:29:57.052 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:29:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 79 M 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 126 M 2024-09-15T03:29:57.053 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 59 MB/s | 29 MB 00:00 2024-09-15T03:29:57.268 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 256 kB/s | 23 kB 00:00 2024-09-15T03:29:57.280 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-09-15T03:29:57.400 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-09-15T03:29:57.959 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 26 MB/s | 35 MB 00:01 2024-09-15T03:29:57.959 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:29:57.959 INFO:teuthology.orchestra.run.smithi078.stdout:Total 50 MB/s | 79 MB 00:01 2024-09-15T03:29:58.122 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:29:58.645 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:29:59.009 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:29:59.009 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:29:59.860 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.8 MB/s | 16 MB 00:02 2024-09-15T03:29:59.969 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:29:59.969 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:30:00.154 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:30:00.154 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:30:00.256 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:30:00.272 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:30:00.272 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-09-15T03:30:00.272 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:30:00.272 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-09-15T03:30:00.273 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:30:00.274 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 79 M 2024-09-15T03:30:00.274 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 126 M 2024-09-15T03:30:00.274 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:30:00.582 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 79 kB/s | 23 kB 00:00 2024-09-15T03:30:00.647 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.7 MB/s | 35 MB 00:06 2024-09-15T03:30:01.464 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.2 MB/s | 29 MB 00:06 2024-09-15T03:30:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:30:01.465 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 79 MB 00:07 2024-09-15T03:30:02.172 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:30:02.416 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:30:02.550 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:30:02.551 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:30:02.743 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-09-15T03:30:03.080 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:03.152 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.9 MB/s | 29 MB 00:05 2024-09-15T03:30:03.152 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:30:03.152 INFO:teuthology.orchestra.run.smithi005.stdout:Total 13 MB/s | 79 MB 00:06 2024-09-15T03:30:03.160 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:03.566 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:30:03.566 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:30:03.860 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:30:04.225 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:30:04.225 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:30:05.218 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:30:05.219 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:30:05.970 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:06.105 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:06.145 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:30:06.665 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.7 MB/s | 29 MB 00:06 2024-09-15T03:30:06.831 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:07.607 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.7 MB/s | 35 MB 00:07 2024-09-15T03:30:07.786 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:30:08.454 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:08.544 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:09.687 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:09.806 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:10.925 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:11.173 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:11.357 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:11.489 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:16.008 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:16.622 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:16.664 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:17.531 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 930 kB/s | 16 MB 00:17 2024-09-15T03:30:17.531 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:30:17.532 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.6 MB/s | 79 MB 00:17 2024-09-15T03:30:18.217 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:30:18.571 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:30:18.571 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:30:19.548 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:30:19.548 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:30:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:21.803 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:22.016 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:30:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:22.748 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:25.711 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:30.578 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:30.628 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:35.447 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:54.250 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:55.392 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:30:55.393 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:30:55.784 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:30:55.864 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:30:55.864 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:30:55.864 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:30:55.864 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:30:55.864 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:30:55.864 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-15T03:30:55.865 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:30:55.865 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:30:55.923 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:30:55.923 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:30:55.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-09-15T03:30:55.955 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-09-15T03:30:55.955 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-09-15T03:30:55.955 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 82.el9 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8274670 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-09-15T03:30:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-09-15T03:30:55.957 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:30:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:30:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:30:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:30:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-09-15T03:30:55.958 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:30:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:30:55.959 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:30:55.959 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:30:55.960 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:30:56.664 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-09-15T03:30:57.982 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:30:58.022 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-09-15T03:30:58.024 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-15T03:30:58.039 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.VrETZLovTr 2024-09-15T03:30:58.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VrETZLovTr 2024-09-15T03:30:58.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.VrETZLovTr 2024-09-15T03:30:58.254 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.VrETZLovTr is 6KB 2024-09-15T03:30:58.305 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.VrETZLovTr 2024-09-15T03:30:58.320 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:30:58.383 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:30:58.383 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:30:58.383 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:30:58.383 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:30:58.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:30:58.531 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-09-15T03:30:58.620 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-09-15T03:30:58.988 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:59.029 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:59.359 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:59.359 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:59.359 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:59.359 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:31:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:31:00.240 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:31:00.620 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:00.710 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:00.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:00.711 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:31:00.711 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:31:00.711 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:31:00.711 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T03:31:00.711 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:31:00.711 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:00.769 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:00.770 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:00.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-09-15T03:31:00.810 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-09-15T03:31:00.810 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 82.el9 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8274670 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:31:00.811 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:31:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:31:00.813 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:31:00.813 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:31:00.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:31:01.091 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.132 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.434 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.434 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:31:01.435 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:31:01.435 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:31:01.477 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-09-15T03:31:02.258 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:02.258 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:31:02.258 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:31:02.258 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.259 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.259 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.259 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.259 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:31:02.259 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:31:02.625 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:02.714 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.715 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:02.715 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:31:02.715 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:31:02.715 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:31:02.716 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T03:31:02.716 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:31:02.716 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:02.774 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.775 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:02.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-09-15T03:31:02.810 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-09-15T03:31:02.810 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.06 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 82.el9 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Group : Unspecified 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 8274670 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:31:02.811 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : builder@centos.org 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:31:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:31:02.814 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:31:02.814 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:31:02.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:31:02.818 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:31:02.849 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-09-15T03:31:02.852 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-15T03:31:02.867 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.qsXLjzkWll 2024-09-15T03:31:02.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qsXLjzkWll 2024-09-15T03:31:02.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.qsXLjzkWll 2024-09-15T03:31:03.067 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.qsXLjzkWll is 6KB 2024-09-15T03:31:03.114 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.qsXLjzkWll 2024-09-15T03:31:03.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:31:03.191 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:31:03.191 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:31:03.191 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:31:03.191 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:31:03.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:31:03.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-09-15T03:31:03.457 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-09-15T03:31:04.811 INFO:teuthology.orchestra.run.smithi005.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:31:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-09-15T03:31:04.843 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-15T03:31:04.877 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.DYHm401oIa 2024-09-15T03:31:04.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DYHm401oIa 2024-09-15T03:31:04.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.DYHm401oIa 2024-09-15T03:31:05.087 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.DYHm401oIa is 6KB 2024-09-15T03:31:05.106 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.DYHm401oIa 2024-09-15T03:31:05.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:31:05.189 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:31:05.189 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:31:05.189 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:31:05.190 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:31:05.191 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:31:05.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-09-15T03:31:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:13.901 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:31:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:31:15.087 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:31:15.088 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:31:15.442 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:15.518 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:15.518 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:15.518 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:31:15.519 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:31:15.519 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:31:15.519 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-15T03:31:15.519 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:31:15.519 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:15.577 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:15.578 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:15.578 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2024-09-15T03:31:15.609 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.06 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 82.el9 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Group : Unspecified 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 8274670 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:31:15.610 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : builder@centos.org 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:31:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:31:15.612 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:31:15.612 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:31:15.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:31:16.302 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-09-15T03:31:17.644 INFO:teuthology.orchestra.run.smithi105.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:31:17.676 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-09-15T03:31:17.678 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-09-15T03:31:17.694 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.DcYHuuHIfI 2024-09-15T03:31:17.695 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DcYHuuHIfI 2024-09-15T03:31:17.763 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.DcYHuuHIfI 2024-09-15T03:31:17.918 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.DcYHuuHIfI is 6KB 2024-09-15T03:31:17.967 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.DcYHuuHIfI 2024-09-15T03:31:17.983 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:31:18.050 INFO:teuthology.orchestra.run.smithi105.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:31:18.051 INFO:teuthology.orchestra.run.smithi105.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:31:18.051 INFO:teuthology.orchestra.run.smithi105.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:31:18.051 INFO:teuthology.orchestra.run.smithi105.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:31:18.052 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:31:18.204 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-09-15T03:31:28.541 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-15T03:31:28.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T03:31:28.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:31:33.349 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T03:31:33.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:31:33.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:31:35.362 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T03:31:35.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-15T03:31:35.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:31:46.881 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-09-15T03:31:48.208 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-15T03:31:48.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-15T03:31:48.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:31:51.745 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-15T03:31:53.789 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-09-15T03:31:55.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T03:31:55.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:31:58.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-09-15T03:32:00.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:32:00.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:02.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-15T03:32:02.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:03.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-15T03:32:06.593 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-09-15T03:32:10.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T03:32:10.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:15.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-15T03:32:15.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:15.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:32:15.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:18.653 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-09-15T03:32:30.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-15T03:32:30.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:34.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-15T03:32:41.635 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-15T03:32:42.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-15T03:32:42.120 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:32:42.121 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-15T03:32:42.137 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:32:42.137 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:32:42.137 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:32:42.137 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:32:42.732 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-15T03:32:43.138 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T03:32:43.138 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:32:43.231 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-09-15T03:32:43.264 DEBUG:teuthology.parallel:result is None 2024-09-15T03:32:43.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T03:32:43.421 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:32:43.422 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-15T03:32:43.438 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:32:43.438 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:32:43.438 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:32:43.438 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:32:44.440 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-15T03:32:44.440 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:32:44.515 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-09-15T03:32:44.545 DEBUG:teuthology.parallel:result is None 2024-09-15T03:32:49.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:32:49.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:49.649 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-15T03:32:50.245 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:32:50.246 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:32:50.246 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-15T03:32:50.288 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:32:50.289 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:32:50.289 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:32:50.289 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:32:51.289 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T03:32:51.290 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:32:51.371 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-09-15T03:32:51.512 DEBUG:teuthology.parallel:result is None 2024-09-15T03:33:02.587 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-09-15T03:33:02.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-15T03:33:02.969 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:33:02.970 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-09-15T03:33:02.984 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:33:02.984 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:33:02.984 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:33:02.985 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:33:03.985 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-15T03:33:03.985 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:33:04.121 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-09-15T03:33:04.150 DEBUG:teuthology.parallel:result is None 2024-09-15T03:33:04.151 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T03:33:04.157 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T03:33:04.158 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:33:04.161 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:33:04.164 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:33:04.167 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:33:04.220 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T03:33:04.227 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T03:33:04.236 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T03:33:04.236 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:33:04.239 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:33:04.241 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:33:04.244 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:33:04.320 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T03:33:04.331 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T03:33:04.332 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:33:04.334 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:33:04.337 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:33:04.339 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:33:04.363 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.369 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.372 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.374 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.388 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.389 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.500 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.532 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:33:04.536 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T03:33:04.543 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T03:33:04.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:33:04.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:33:04.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:33:04.550 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:33:04.608 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T03:33:04.618 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T03:33:04.618 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:33:04.621 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:33:04.623 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:33:04.626 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:33:04.666 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:04.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:04.774 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:04.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:04.922 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:33:04.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:33:04.989 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:05.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:05.103 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:05.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:05.241 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:33:05.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:33:05.319 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:05.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:05.423 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:05.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:05.553 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:33:05.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:33:05.619 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:05.664 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:33:05.713 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:05.796 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:33:05.839 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:33:05.839 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:33:05.904 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-09-15T03:33:05.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-09-15T03:33:05.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-09-15T03:33:05.913 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-09-15T03:33:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:33:05.966 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:33:05.970 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:33:05.995 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:33:06.495 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T03:33:07.344 INFO:teuthology.task.internal:Starting timer... 2024-09-15T03:33:07.344 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T03:33:07.829 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T03:33:08.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-09-15T03:33:08.083 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-09-15T03:33:08.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-09-15T03:33:08.143 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-09-15T03:33:08.145 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-09-15T03:33:08.197 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-09-15T03:33:08.199 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-09-15T03:33:08.242 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-09-15T03:33:08.244 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T03:33:08.244 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-09-15T03:33:08.273 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-09-15T03:33:08.274 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-09-15T03:33:08.303 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-09-15T03:33:08.303 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-09-15T03:33:08.320 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-09-15T03:33:08.321 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-09-15T03:33:08.341 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-09-15T03:33:08.341 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive'} 2024-09-15T03:33:08.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:33:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:33:08.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:33:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:33:08.395 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:33:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:33:08.421 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:33:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:33:08.446 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T03:33:08.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:33:08.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:33:08.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:33:08.544 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:33:08.573 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T03:33:08.583 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T03:33:08.597 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T03:33:08.601 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi064.front.sepia.ceph.com,smithi078.front.sepia.ceph.com,smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T03:37:00.941 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2024-09-15T03:37:00.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-15T03:37:00.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:37:01.045 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-15T03:37:01.135 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-15T03:37:01.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:37:01.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:37:01.222 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-15T03:37:01.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-15T03:37:01.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T03:37:01.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:37:01.393 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-15T03:37:01.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T03:37:01.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-15T03:37:01.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:37:01.609 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-09-15T03:37:01.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-15T03:37:01.709 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T03:37:02.458 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T03:37:02.459 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:37:02.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:37:02.462 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:37:02.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:37:02.466 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:37:02.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:37:02.469 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:37:02.469 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:37:02.494 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:37:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:37:02.500 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:37:02.502 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:37:02.511 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:37:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:37:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:37:02.518 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:37:02.540 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-09-15T03:37:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-09-15T03:37:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-09-15T03:37:02.545 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-09-15T03:37:02.551 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-09-15T03:37:02.553 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-09-15T03:37:02.557 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-09-15T03:37:02.557 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-09-15T03:37:02.567 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:37:02.568 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:37:02.571 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:37:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:37:02.583 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:37:02.583 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:37:02.586 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:37:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:37:02.641 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:37:02.642 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:37:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:37:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-09-15T03:37:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.644 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.645 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:37:02.645 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-09-15T03:37:02.645 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.646 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.646 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.646 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.648 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:37:02.651 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:37:02.651 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-15T03:37:02.651 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.651 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.651 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.652 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.652 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:37:02.655 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:37:02.655 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-09-15T03:37:02.655 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.655 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.656 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.656 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:37:02.656 INFO:teuthology.run_tasks:Running task install... 2024-09-15T03:37:02.663 DEBUG:teuthology.task.install:project ceph 2024-09-15T03:37:02.664 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:37:02.664 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T03:37:02.664 INFO:teuthology.task.install:Using flavor: default 2024-09-15T03:37:02.670 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T03:37:02.670 INFO:teuthology.task.install:extra packages: [] 2024-09-15T03:37:02.670 DEBUG:teuthology.orchestra.run.smithi005:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:37:02.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:37:02.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:37:02.671 DEBUG:teuthology.orchestra.run.smithi105:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:37:03.241 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:36:09 AM UTC. 2024-09-15T03:37:03.244 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:36:09 AM UTC. 2024-09-15T03:37:03.274 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:48 ago on Sun 15 Sep 2024 03:36:15 AM UTC. 2024-09-15T03:37:03.276 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:48 ago on Sun 15 Sep 2024 03:36:15 AM UTC. 2024-09-15T03:37:03.327 INFO:teuthology.orchestra.run.smithi105.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:37:03.329 INFO:teuthology.orchestra.run.smithi078.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:37:03.364 INFO:teuthology.orchestra.run.smithi005.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:37:03.365 INFO:teuthology.orchestra.run.smithi064.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:37:03.369 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:37:03.370 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:37:03.370 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:37:03.371 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:37:03.372 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:37:03.372 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:37:03.409 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:37:03.410 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:37:03.410 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:37:03.410 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:37:03.411 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:37:03.411 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:37:03.432 DEBUG:teuthology.orchestra.run.smithi105:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:37:03.433 DEBUG:teuthology.orchestra.run.smithi078:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:37:03.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:37:03.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:of the main distribution, and quality may vary. 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:, 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:and packages are not held to any quality or security level. 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:37:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:37:03.610 INFO:teuthology.orchestra.run.smithi105.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:of the main distribution, and quality may vary. 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:, 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:and packages are not held to any quality or security level. 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:37:03.611 INFO:teuthology.orchestra.run.smithi105.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:of the main distribution, and quality may vary. 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:, 2024-09-15T03:37:03.647 INFO:teuthology.orchestra.run.smithi064.stderr:and packages are not held to any quality or security level. 2024-09-15T03:37:03.648 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-15T03:37:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:37:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:37:03.658 INFO:teuthology.orchestra.run.smithi005.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:37:03.658 INFO:teuthology.orchestra.run.smithi005.stderr:of the main distribution, and quality may vary. 2024-09-15T03:37:03.658 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:37:03.658 INFO:teuthology.orchestra.run.smithi005.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:37:03.659 INFO:teuthology.orchestra.run.smithi005.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:37:03.659 INFO:teuthology.orchestra.run.smithi005.stderr:, 2024-09-15T03:37:03.659 INFO:teuthology.orchestra.run.smithi005.stderr:and packages are not held to any quality or security level. 2024-09-15T03:37:03.659 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:37:03.659 INFO:teuthology.orchestra.run.smithi005.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:37:03.659 INFO:teuthology.orchestra.run.smithi005.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:37:03.668 INFO:teuthology.orchestra.run.smithi078.stdout:Repository successfully enabled. 2024-09-15T03:37:03.670 INFO:teuthology.orchestra.run.smithi105.stdout:Repository successfully enabled. 2024-09-15T03:37:03.704 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:37:03.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:37:03.710 INFO:teuthology.orchestra.run.smithi064.stdout:Repository successfully enabled. 2024-09-15T03:37:03.712 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:37:03.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:37:03.725 INFO:teuthology.orchestra.run.smithi005.stdout:Repository successfully enabled. 2024-09-15T03:37:03.754 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:37:03.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:37:03.761 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:37:03.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:37:03.868 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:37:03.868 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:37:03.873 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:37:03.873 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:37:03.909 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:37:03.909 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:37:03.913 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:37:03.913 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:37:03.983 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:37:03.983 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:37:03.983 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:37:03.989 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:37:03.989 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:37:03.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:37:04.009 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:37:04.010 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:37:04.017 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:37:04.017 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:37:04.025 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:37:04.025 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:37:04.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:37:04.032 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:37:04.032 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:37:04.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:37:04.052 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:37:04.052 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:37:04.059 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:37:04.060 DEBUG:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:37:04.076 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:37:04.082 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:37:04.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:37:04.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:37:04.152 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:37:04.157 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:37:04.195 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:37:04.202 DEBUG:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:37:04.216 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-09-15T03:37:04.217 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-09-15T03:37:04.221 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-09-15T03:37:04.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-09-15T03:37:04.260 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-09-15T03:37:04.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-09-15T03:37:04.268 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2024-09-15T03:37:04.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum clean all 2024-09-15T03:37:04.447 INFO:teuthology.orchestra.run.smithi078.stdout:45 files removed 2024-09-15T03:37:04.450 INFO:teuthology.orchestra.run.smithi105.stdout:45 files removed 2024-09-15T03:37:04.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw 2024-09-15T03:37:04.476 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2024-09-15T03:37:04.494 INFO:teuthology.orchestra.run.smithi064.stdout:45 files removed 2024-09-15T03:37:04.517 INFO:teuthology.orchestra.run.smithi005.stdout:45 files removed 2024-09-15T03:37:04.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-09-15T03:37:04.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw 2024-09-15T03:37:05.026 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 290 kB/s | 77 kB 00:00 2024-09-15T03:37:05.035 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 282 kB/s | 77 kB 00:00 2024-09-15T03:37:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 280 kB/s | 77 kB 00:00 2024-09-15T03:37:05.113 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 288 kB/s | 77 kB 00:00 2024-09-15T03:37:05.318 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-15T03:37:05.327 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-15T03:37:05.380 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T03:37:05.397 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T03:37:05.594 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T03:37:05.594 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-15T03:37:05.656 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T03:37:05.656 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 8.1 kB/s | 1.8 kB 00:00 2024-09-15T03:37:05.745 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for el9 owned by ceph 66 kB/s | 7.6 kB 00:00 2024-09-15T03:37:05.746 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for el9 owned by ceph 66 kB/s | 7.6 kB 00:00 2024-09-15T03:37:05.798 INFO:teuthology.orchestra.run.smithi005.stdout:Copr repo for el9 owned by ceph 72 kB/s | 7.6 kB 00:00 2024-09-15T03:37:05.815 INFO:teuthology.orchestra.run.smithi064.stdout:Copr repo for el9 owned by ceph 63 kB/s | 7.6 kB 00:00 2024-09-15T03:37:06.324 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-15T03:37:07.204 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.2 MB 00:01 2024-09-15T03:37:07.599 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-09-15T03:37:09.120 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.2 MB 00:03 2024-09-15T03:37:09.411 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-15T03:37:09.642 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 20 MB 00:02 2024-09-15T03:37:11.046 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T03:37:11.796 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-09-15T03:37:13.878 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T03:37:14.967 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-09-15T03:37:15.314 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T03:37:15.529 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 19 kB 00:00 2024-09-15T03:37:16.139 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-15T03:37:16.375 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-15T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 19 kB 00:00 2024-09-15T03:37:17.048 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-15T03:37:17.798 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-15T03:37:17.914 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-15T03:37:17.952 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-15T03:37:18.727 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-15T03:37:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-15T03:37:23.367 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T03:37:23.662 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-15T03:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-15T03:37:24.669 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:37:24.687 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:37:24.688 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:24.689 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:37:24.690 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:37:24.690 INFO:teuthology.orchestra.run.smithi005.stdout:Install 22 Packages 2024-09-15T03:37:24.690 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 2 Packages 2024-09-15T03:37:24.690 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:24.690 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 58 M 2024-09-15T03:37:24.691 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:37:25.699 INFO:teuthology.orchestra.run.smithi005.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 5.9 MB/s | 5.0 MB 00:00 2024-09-15T03:37:25.749 INFO:teuthology.orchestra.run.smithi005.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 506 kB/s | 25 kB 00:00 2024-09-15T03:37:25.884 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:37:25.887 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:37:25.888 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout:Install 22 Packages 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-09-15T03:37:25.889 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:25.890 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 58 M 2024-09-15T03:37:25.890 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:37:25.975 INFO:teuthology.orchestra.run.smithi005.stdout:(3/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 20 MB/s | 22 MB 00:01 2024-09-15T03:37:26.008 INFO:teuthology.orchestra.run.smithi005.stdout:(4/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 2.4 MB/s | 637 kB 00:00 2024-09-15T03:37:26.150 INFO:teuthology.orchestra.run.smithi005.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 11 MB/s | 14 MB 00:01 2024-09-15T03:37:26.176 INFO:teuthology.orchestra.run.smithi005.stdout:(6/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.2 MB/s | 458 kB 00:00 2024-09-15T03:37:26.201 INFO:teuthology.orchestra.run.smithi005.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 902 kB/s | 45 kB 00:00 2024-09-15T03:37:26.226 INFO:teuthology.orchestra.run.smithi005.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.3 MB/s | 120 kB 00:00 2024-09-15T03:37:26.248 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:37:26.251 INFO:teuthology.orchestra.run.smithi005.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.0 MB/s | 156 kB 00:00 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-09-15T03:37:26.261 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:37:26.262 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:37:26.263 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout:Install 22 Packages 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:26.264 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 58 M 2024-09-15T03:37:26.265 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:37:26.285 INFO:teuthology.orchestra.run.smithi005.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 5.3 MB/s | 320 kB 00:00 2024-09-15T03:37:26.311 INFO:teuthology.orchestra.run.smithi005.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-15T03:37:26.344 INFO:teuthology.orchestra.run.smithi005.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.7 MB/s | 100 kB 00:00 2024-09-15T03:37:26.436 INFO:teuthology.orchestra.run.smithi005.stdout:(13/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 4.9 MB 00:00 2024-09-15T03:37:26.545 INFO:teuthology.orchestra.run.smithi005.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 685 kB/s | 160 kB 00:00 2024-09-15T03:37:26.570 INFO:teuthology.orchestra.run.smithi005.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 200 kB/s | 45 kB 00:00 2024-09-15T03:37:26.618 INFO:teuthology.orchestra.run.smithi105.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-15T03:37:26.620 INFO:teuthology.orchestra.run.smithi005.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-15T03:37:26.654 INFO:teuthology.orchestra.run.smithi005.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-15T03:37:26.660 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:37:26.667 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:37:26.667 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:37:26.667 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:37:26.667 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:37:26.667 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:37:26.667 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:37:26.668 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:37:26.669 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout:Install 22 Packages 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 2 Packages 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 58 M 2024-09-15T03:37:26.670 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:37:26.676 INFO:teuthology.orchestra.run.smithi105.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 433 kB/s | 25 kB 00:00 2024-09-15T03:37:26.712 INFO:teuthology.orchestra.run.smithi005.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-09-15T03:37:26.746 INFO:teuthology.orchestra.run.smithi005.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 586 kB/s | 49 kB 00:00 2024-09-15T03:37:26.780 INFO:teuthology.orchestra.run.smithi005.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2024-09-15T03:37:26.785 INFO:teuthology.orchestra.run.smithi105.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 5.7 MB/s | 637 kB 00:00 2024-09-15T03:37:26.821 INFO:teuthology.orchestra.run.smithi005.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 631 kB/s | 67 kB 00:00 2024-09-15T03:37:26.881 INFO:teuthology.orchestra.run.smithi005.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-15T03:37:26.885 INFO:teuthology.orchestra.run.smithi105.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 4.5 MB/s | 458 kB 00:00 2024-09-15T03:37:27.205 INFO:teuthology.orchestra.run.smithi005.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 7.2 MB/s | 3.1 MB 00:00 2024-09-15T03:37:27.297 INFO:teuthology.orchestra.run.smithi005.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 6.2 MB/s | 2.9 MB 00:00 2024-09-15T03:37:27.303 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:27.303 INFO:teuthology.orchestra.run.smithi005.stdout:Total 22 MB/s | 58 MB 00:02 2024-09-15T03:37:27.361 INFO:teuthology.orchestra.run.smithi105.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 12 MB/s | 14 MB 00:01 2024-09-15T03:37:27.369 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:37:27.407 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:37:27.408 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:37:27.670 INFO:teuthology.orchestra.run.smithi105.stdout:(6/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 15 MB/s | 22 MB 00:01 2024-09-15T03:37:27.736 INFO:teuthology.orchestra.run.smithi105.stdout:(7/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 5.8 MB/s | 4.9 MB 00:00 2024-09-15T03:37:27.762 INFO:teuthology.orchestra.run.smithi105.stdout:(8/24): python3-ceph-argparse-17.2.7-1643.gd75f 113 kB/s | 45 kB 00:00 2024-09-15T03:37:27.787 INFO:teuthology.orchestra.run.smithi105.stdout:(9/24): python3-ceph-common-17.2.7-1643.gd75f23 1.0 MB/s | 120 kB 00:00 2024-09-15T03:37:27.812 INFO:teuthology.orchestra.run.smithi105.stdout:(10/24): python3-cephfs-17.2.7-1643.gd75f2382.e 2.0 MB/s | 156 kB 00:00 2024-09-15T03:37:27.837 INFO:teuthology.orchestra.run.smithi105.stdout:(11/24): python3-rados-17.2.7-1643.gd75f2382.el 4.1 MB/s | 320 kB 00:00 2024-09-15T03:37:27.862 INFO:teuthology.orchestra.run.smithi105.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T03:37:27.896 INFO:teuthology.orchestra.run.smithi105.stdout:(13/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 2.6 MB/s | 295 kB 00:00 2024-09-15T03:37:27.904 INFO:teuthology.orchestra.run.smithi064.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 3.2 MB/s | 5.0 MB 00:01 2024-09-15T03:37:27.909 INFO:teuthology.orchestra.run.smithi078.stdout:(1/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 15 MB/s | 14 MB 00:00 2024-09-15T03:37:27.946 INFO:teuthology.orchestra.run.smithi064.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 609 kB/s | 25 kB 00:00 2024-09-15T03:37:27.955 INFO:teuthology.orchestra.run.smithi105.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 492 kB/s | 45 kB 00:00 2024-09-15T03:37:27.959 INFO:teuthology.orchestra.run.smithi078.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 506 kB/s | 25 kB 00:00 2024-09-15T03:37:27.989 INFO:teuthology.orchestra.run.smithi105.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-15T03:37:28.018 INFO:teuthology.orchestra.run.smithi078.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 11 MB/s | 637 kB 00:00 2024-09-15T03:37:28.030 INFO:teuthology.orchestra.run.smithi064.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-15T03:37:28.039 INFO:teuthology.orchestra.run.smithi105.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-09-15T03:37:28.064 INFO:teuthology.orchestra.run.smithi105.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-09-15T03:37:28.077 INFO:teuthology.orchestra.run.smithi078.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 7.7 MB/s | 458 kB 00:00 2024-09-15T03:37:28.114 INFO:teuthology.orchestra.run.smithi105.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-15T03:37:28.117 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:37:28.117 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:37:28.130 INFO:teuthology.orchestra.run.smithi064.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 4.5 MB/s | 458 kB 00:00 2024-09-15T03:37:28.148 INFO:teuthology.orchestra.run.smithi105.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-09-15T03:37:28.181 INFO:teuthology.orchestra.run.smithi105.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-15T03:37:28.207 INFO:teuthology.orchestra.run.smithi105.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 747 kB/s | 67 kB 00:00 2024-09-15T03:37:28.227 INFO:teuthology.orchestra.run.smithi078.stdout:(5/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 3.8 MB/s | 5.0 MB 00:01 2024-09-15T03:37:28.276 INFO:teuthology.orchestra.run.smithi105.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T03:37:28.302 INFO:teuthology.orchestra.run.smithi078.stdout:(6/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 22 MB/s | 4.9 MB 00:00 2024-09-15T03:37:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 450 kB/s | 45 kB 00:00 2024-09-15T03:37:28.386 INFO:teuthology.orchestra.run.smithi078.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 1.4 MB/s | 120 kB 00:00 2024-09-15T03:37:28.420 INFO:teuthology.orchestra.run.smithi078.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.7 MB/s | 156 kB 00:00 2024-09-15T03:37:28.487 INFO:teuthology.orchestra.run.smithi078.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 3.1 MB/s | 320 kB 00:00 2024-09-15T03:37:28.512 INFO:teuthology.orchestra.run.smithi078.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 3.1 MB/s | 295 kB 00:00 2024-09-15T03:37:28.533 INFO:teuthology.orchestra.run.smithi105.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 9.0 MB/s | 2.9 MB 00:00 2024-09-15T03:37:28.554 INFO:teuthology.orchestra.run.smithi078.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.5 MB/s | 100 kB 00:00 2024-09-15T03:37:28.591 INFO:teuthology.orchestra.run.smithi105.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 8.0 MB/s | 3.1 MB 00:00 2024-09-15T03:37:28.596 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:28.597 INFO:teuthology.orchestra.run.smithi105.stdout:Total 21 MB/s | 58 MB 00:02 2024-09-15T03:37:28.612 INFO:teuthology.orchestra.run.smithi078.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-09-15T03:37:28.638 INFO:teuthology.orchestra.run.smithi078.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 540 kB/s | 45 kB 00:00 2024-09-15T03:37:28.660 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:37:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-15T03:37:28.693 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:37:28.694 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:37:28.730 INFO:teuthology.orchestra.run.smithi078.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-15T03:37:28.739 INFO:teuthology.orchestra.run.smithi064.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 8.1 MB/s | 4.9 MB 00:00 2024-09-15T03:37:28.755 INFO:teuthology.orchestra.run.smithi078.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 633 kB/s | 42 kB 00:00 2024-09-15T03:37:28.789 INFO:teuthology.orchestra.run.smithi064.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 907 kB/s | 45 kB 00:00 2024-09-15T03:37:28.806 INFO:teuthology.orchestra.run.smithi078.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-15T03:37:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.1 MB/s | 67 kB 00:00 2024-09-15T03:37:28.865 INFO:teuthology.orchestra.run.smithi078.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-09-15T03:37:28.932 INFO:teuthology.orchestra.run.smithi078.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-15T03:37:28.956 INFO:teuthology.orchestra.run.smithi064.stdout:(7/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 5.5 MB/s | 14 MB 00:02 2024-09-15T03:37:28.981 INFO:teuthology.orchestra.run.smithi064.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 626 kB/s | 120 kB 00:00 2024-09-15T03:37:29.031 INFO:teuthology.orchestra.run.smithi064.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.0 MB/s | 156 kB 00:00 2024-09-15T03:37:29.082 INFO:teuthology.orchestra.run.smithi064.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 3.1 MB/s | 320 kB 00:00 2024-09-15T03:37:29.124 INFO:teuthology.orchestra.run.smithi078.stdout:(22/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-09-15T03:37:29.140 INFO:teuthology.orchestra.run.smithi064.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 2.7 MB/s | 295 kB 00:00 2024-09-15T03:37:29.166 INFO:teuthology.orchestra.run.smithi064.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.2 MB/s | 100 kB 00:00 2024-09-15T03:37:29.274 INFO:teuthology.orchestra.run.smithi064.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-15T03:37:29.299 INFO:teuthology.orchestra.run.smithi064.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-15T03:37:29.312 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:37:29.312 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:37:29.408 INFO:teuthology.orchestra.run.smithi064.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-15T03:37:29.449 INFO:teuthology.orchestra.run.smithi078.stdout:(23/24): ceph-common-17.2.7-1643.gd75f2382.el9. 8.9 MB/s | 22 MB 00:02 2024-09-15T03:37:29.458 INFO:teuthology.orchestra.run.smithi064.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-15T03:37:29.508 INFO:teuthology.orchestra.run.smithi078.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-09-15T03:37:29.508 INFO:teuthology.orchestra.run.smithi064.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-15T03:37:29.509 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:37:29.513 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:29.513 INFO:teuthology.orchestra.run.smithi078.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-15T03:37:29.534 INFO:teuthology.orchestra.run.smithi064.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 662 kB 00:00 2024-09-15T03:37:29.559 INFO:teuthology.orchestra.run.smithi064.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-15T03:37:29.575 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:37:29.607 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:37:29.607 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:37:29.609 INFO:teuthology.orchestra.run.smithi064.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-09-15T03:37:29.660 INFO:teuthology.orchestra.run.smithi064.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-15T03:37:29.898 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:37:29.910 INFO:teuthology.orchestra.run.smithi064.stdout:(22/24): librados2-17.2.7-1643.gd75f2382.el9.x8 10 MB/s | 3.1 MB 00:00 2024-09-15T03:37:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 9.8 MB/s | 2.9 MB 00:00 2024-09-15T03:37:29.966 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:30.060 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:30.148 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:30.176 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:30.265 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:30.277 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:37:30.277 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:37:30.311 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:37:30.340 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:37:30.535 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:37:30.804 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:37:30.846 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:30.870 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:37:30.917 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:30.963 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:37:30.985 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:31.015 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:37:31.052 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:31.077 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:31.115 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:31.141 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:31.190 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:37:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:37:31.292 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:37:31.346 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:37:31.383 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:37:31.528 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:37:31.634 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:37:31.678 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:31.753 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:31.820 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:37:31.829 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:37:31.848 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:31.855 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:37:31.864 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:37:31.910 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:31.947 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:37:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:31.999 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:37:32.039 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:32.043 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:32.107 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:32.122 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:32.134 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:32.145 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:32.200 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:32.231 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:32.240 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:37:32.284 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:37:32.292 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:37:32.317 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:37:32.324 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:37:32.609 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:37:32.655 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:32.737 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:32.762 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:37:32.789 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:32.818 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:37:32.835 INFO:teuthology.orchestra.run.smithi064.stdout:(24/24): ceph-common-17.2.7-1643.gd75f2382.el9. 3.5 MB/s | 22 MB 00:06 2024-09-15T03:37:32.841 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:32.841 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.8 MB/s | 58 MB 00:06 2024-09-15T03:37:32.849 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:37:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:32.880 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:37:32.908 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:37:32.923 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:37:32.946 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:37:32.946 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:37:32.960 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:33.116 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:33.169 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:37:33.208 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:37:33.241 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:37:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:37:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:33.592 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:37:33.592 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:37:33.668 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:33.742 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:37:33.772 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:37:34.359 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:34.394 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:37:34.944 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:34.989 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:35.100 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:35.100 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:37:35.100 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:37:35.100 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:35.141 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:35.227 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:37:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:35.298 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:35.304 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:37:35.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:37:35.842 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:35.882 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:35.991 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:35.992 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:37:35.992 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:37:35.992 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:36.025 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:36.216 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:37:36.252 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:37:36.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:36.371 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:36.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:36.463 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:36.520 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:37:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:37:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:37:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:36.830 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:37:36.922 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:36.923 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:37:36.923 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:37:36.923 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:36.942 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:36.956 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:37.110 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:37:37.136 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:37:39.634 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:39.666 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:40.235 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:40.269 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:40.363 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:40.363 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:37:40.364 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:37:40.364 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:45.593 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /root 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2024-09-15T03:37:45.594 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:45.880 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-09-15T03:37:45.881 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-09-15T03:37:46.635 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-09-15T03:37:46.636 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:46.970 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:47.049 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:47.050 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:47.050 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:37:47.050 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:47.050 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:47.050 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:47.051 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:47.140 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:47.140 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:47.227 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:47.306 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:47.306 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:37:47.306 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:47.306 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:47.388 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:47.389 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:47.930 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:48.009 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:48.009 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:37:48.009 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:48.009 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:48.099 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:48.100 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:37:48.432 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:37:48.433 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:37:48.434 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:48.434 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:37:48.434 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:37:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:37:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:37:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:37:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:49.228 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:37:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:37:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:37:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:37:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:37:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:37:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:37:49.641 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-test 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.713 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:37:49.714 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:49.715 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:37:50.103 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2024-09-15T03:37:50.224 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:37:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:50.342 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:37:50.362 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:37:50.362 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:37:50.362 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-09-15T03:37:50.363 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:37:50.364 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 48 M 2024-09-15T03:37:50.365 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 193 M 2024-09-15T03:37:50.365 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:37:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-09-15T03:37:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-09-15T03:37:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-09-15T03:37:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-09-15T03:37:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-09-15T03:37:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-09-15T03:37:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-09-15T03:37:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:50.670 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:37:50.687 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-15T03:37:50.712 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 662 kB/s | 163 kB 00:00 2024-09-15T03:37:50.738 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T03:37:50.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-test 2024-09-15T03:37:50.807 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:37:50.807 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:37:50.807 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:37:50.807 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:37:50.807 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-09-15T03:37:50.808 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:37:50.809 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 48 M 2024-09-15T03:37:50.809 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 193 M 2024-09-15T03:37:50.809 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:37:51.218 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-15T03:37:51.294 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-09-15T03:37:51.331 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:37:51.468 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:37:51.468 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:37:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:37:51.468 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:37:51.468 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:37:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-09-15T03:37:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:37:51.470 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 48 M 2024-09-15T03:37:51.470 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 193 M 2024-09-15T03:37:51.470 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:37:51.477 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 394 kB/s | 163 kB 00:00 2024-09-15T03:37:51.746 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.7 MB/s | 304 kB 00:00 2024-09-15T03:37:51.771 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 807 kB/s | 163 kB 00:00 2024-09-15T03:37:51.821 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 847 kB/s | 64 kB 00:00 2024-09-15T03:37:51.988 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-15T03:37:51.990 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:51.990 INFO:teuthology.orchestra.run.smithi005.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-15T03:37:52.019 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:37:52.022 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:52.040 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:37:52.040 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:37:52.085 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:52.085 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:52.085 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:37:52.086 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:52.086 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:52.086 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:52.087 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:52.428 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:37:52.428 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:37:52.978 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:37:53.066 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:37:53.105 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:37:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:53.179 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 48 MB 00:01 2024-09-15T03:37:53.181 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:53.182 INFO:teuthology.orchestra.run.smithi078.stdout:Total 28 MB/s | 48 MB 00:01 2024-09-15T03:37:53.208 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:37:53.227 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:37:53.227 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:37:53.592 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:37:53.592 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:37:53.902 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 17 MB/s | 48 MB 00:02 2024-09-15T03:37:53.904 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:53.904 INFO:teuthology.orchestra.run.smithi105.stdout:Total 16 MB/s | 48 MB 00:03 2024-09-15T03:37:53.932 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:37:53.951 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:37:53.951 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:37:54.107 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:37:54.184 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:37:54.232 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:37:54.256 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:54.316 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:37:54.317 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:37:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:54.469 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:37:54.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-09-15T03:37:54.846 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:37:54.929 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:37:54.968 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:37:55.004 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:55.380 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:32 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:37:55.527 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:37:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 48 M 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 193 M 2024-09-15T03:37:55.529 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:37:55.746 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.7 MB/s | 304 kB 00:00 2024-09-15T03:37:55.788 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-15T03:37:55.830 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 640 kB/s | 163 kB 00:00 2024-09-15T03:37:57.114 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-15T03:37:57.116 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:57.116 INFO:teuthology.orchestra.run.smithi064.stdout:Total 31 MB/s | 48 MB 00:01 2024-09-15T03:37:57.144 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:37:57.165 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:37:57.166 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:37:57.550 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:37:57.551 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:37:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:37:58.094 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:37:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:37:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:37:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:58.671 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:58.746 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:37:59.443 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:59.519 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:37:59.942 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:37:59.942 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:37:59.942 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:37:59.943 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:38:00.274 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:38:00.275 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:00.275 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:00.275 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:00.275 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:00.276 INFO:teuthology.orchestra.run.smithi005.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:38:00.276 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:38:00.276 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:00.276 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:00.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph 2024-09-15T03:38:00.941 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:38:00.941 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:38:00.942 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:38:00.942 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:38:01.163 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:01.262 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:38:01.262 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:01.262 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:01.262 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:01.262 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:01.262 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:38:01.263 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:38:01.263 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:01.263 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:01.303 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:38:01.306 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections 2024-09-15T03:38:01.307 INFO:teuthology.orchestra.run.smithi005.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:38:01.308 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:01.309 INFO:teuthology.orchestra.run.smithi005.stdout:Install 34 Packages 2024-09-15T03:38:01.310 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:01.310 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 27 M 2024-09-15T03:38:01.310 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 96 M 2024-09-15T03:38:01.310 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:01.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph 2024-09-15T03:38:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:38:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:38:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:38:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:38:01.856 INFO:teuthology.orchestra.run.smithi005.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-15T03:38:02.056 INFO:teuthology.orchestra.run.smithi005.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-15T03:38:02.107 INFO:teuthology.orchestra.run.smithi005.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-15T03:38:02.110 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:38 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:38:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:02.147 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:38:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:38:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:02.148 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:02.182 INFO:teuthology.orchestra.run.smithi005.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-15T03:38:02.207 INFO:teuthology.orchestra.run.smithi005.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.3 MB/s | 240 kB 00:00 2024-09-15T03:38:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:38:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:38:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout:Install 34 Packages 2024-09-15T03:38:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:02.255 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 27 M 2024-09-15T03:38:02.255 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 96 M 2024-09-15T03:38:02.255 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:02.316 INFO:teuthology.orchestra.run.smithi005.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-15T03:38:02.358 INFO:teuthology.orchestra.run.smithi005.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T03:38:02.400 INFO:teuthology.orchestra.run.smithi005.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 6.4 MB/s | 1.2 MB 00:00 2024-09-15T03:38:02.425 INFO:teuthology.orchestra.run.smithi005.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-15T03:38:02.436 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2024-09-15T03:38:02.450 INFO:teuthology.orchestra.run.smithi005.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-15T03:38:02.475 INFO:teuthology.orchestra.run.smithi005.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-15T03:38:02.651 INFO:teuthology.orchestra.run.smithi005.stdout:(12/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 26 MB/s | 15 MB 00:00 2024-09-15T03:38:02.763 INFO:teuthology.orchestra.run.smithi078.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-15T03:38:02.859 INFO:teuthology.orchestra.run.smithi005.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 90 kB/s | 35 kB 00:00 2024-09-15T03:38:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:38:02.980 INFO:teuthology.orchestra.run.smithi078.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-09-15T03:38:03.008 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:40 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:03.014 INFO:teuthology.orchestra.run.smithi078.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.7 MB/s | 1.4 MB 00:00 2024-09-15T03:38:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:38:03.021 INFO:teuthology.orchestra.run.smithi005.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 208 kB/s | 77 kB 00:00 2024-09-15T03:38:03.055 INFO:teuthology.orchestra.run.smithi005.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 261 kB/s | 51 kB 00:00 2024-09-15T03:38:03.097 INFO:teuthology.orchestra.run.smithi005.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-15T03:38:03.106 INFO:teuthology.orchestra.run.smithi078.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.6 MB/s | 240 kB 00:00 2024-09-15T03:38:03.131 INFO:teuthology.orchestra.run.smithi005.stdout:(17/34): python3-mako-1.1.4-6.el9.noarch.rpm 253 kB/s | 172 kB 00:00 2024-09-15T03:38:03.156 INFO:teuthology.orchestra.run.smithi005.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 309 kB/s | 42 kB 00:00 2024-09-15T03:38:03.175 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:03.177 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:03.177 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:03.177 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:03.177 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:38:03.178 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:38:03.179 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:38:03.180 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:03.181 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:03.182 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:03.182 INFO:teuthology.orchestra.run.smithi105.stdout:Install 34 Packages 2024-09-15T03:38:03.182 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:03.182 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 27 M 2024-09-15T03:38:03.182 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 96 M 2024-09-15T03:38:03.182 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:03.198 INFO:teuthology.orchestra.run.smithi005.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-15T03:38:03.223 INFO:teuthology.orchestra.run.smithi005.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 181 kB/s | 11 kB 00:00 2024-09-15T03:38:03.239 INFO:teuthology.orchestra.run.smithi078.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-15T03:38:03.249 INFO:teuthology.orchestra.run.smithi005.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 366 kB/s | 18 kB 00:00 2024-09-15T03:38:03.274 INFO:teuthology.orchestra.run.smithi005.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-15T03:38:03.300 INFO:teuthology.orchestra.run.smithi005.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-15T03:38:03.325 INFO:teuthology.orchestra.run.smithi005.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-15T03:38:03.348 INFO:teuthology.orchestra.run.smithi078.stdout:(6/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 6.7 MB/s | 3.9 MB 00:00 2024-09-15T03:38:03.350 INFO:teuthology.orchestra.run.smithi005.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-15T03:38:03.375 INFO:teuthology.orchestra.run.smithi005.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:38:03.409 INFO:teuthology.orchestra.run.smithi005.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-15T03:38:03.434 INFO:teuthology.orchestra.run.smithi005.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-15T03:38:03.459 INFO:teuthology.orchestra.run.smithi005.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-15T03:38:03.485 INFO:teuthology.orchestra.run.smithi005.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-15T03:38:03.506 INFO:teuthology.orchestra.run.smithi078.stdout:(7/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-15T03:38:03.509 INFO:teuthology.orchestra.run.smithi005.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-15T03:38:03.535 INFO:teuthology.orchestra.run.smithi005.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-15T03:38:03.548 INFO:teuthology.orchestra.run.smithi078.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2024-09-15T03:38:03.560 INFO:teuthology.orchestra.run.smithi005.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-15T03:38:03.585 INFO:teuthology.orchestra.run.smithi005.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-15T03:38:03.590 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:03.590 INFO:teuthology.orchestra.run.smithi005.stdout:Total 12 MB/s | 27 MB 00:02 2024-09-15T03:38:03.591 INFO:teuthology.orchestra.run.smithi078.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-15T03:38:03.616 INFO:teuthology.orchestra.run.smithi078.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 398 kB/s | 106 kB 00:00 2024-09-15T03:38:03.641 INFO:teuthology.orchestra.run.smithi078.stdout:(11/34): python3-pycparser-2.20-6.el9.noarch.rp 1.0 MB/s | 135 kB 00:00 2024-09-15T03:38:03.666 INFO:teuthology.orchestra.run.smithi078.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-15T03:38:03.728 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:03.750 INFO:teuthology.orchestra.run.smithi078.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-15T03:38:03.769 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:03.770 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:03.800 INFO:teuthology.orchestra.run.smithi078.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 577 kB/s | 77 kB 00:00 2024-09-15T03:38:03.825 INFO:teuthology.orchestra.run.smithi078.stdout:(15/34): python3-mako-1.1.4-6.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-09-15T03:38:03.859 INFO:teuthology.orchestra.run.smithi078.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 467 kB/s | 51 kB 00:00 2024-09-15T03:38:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 496 kB/s | 42 kB 00:00 2024-09-15T03:38:03.918 INFO:teuthology.orchestra.run.smithi078.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-15T03:38:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 434 kB/s | 11 kB 00:00 2024-09-15T03:38:03.977 INFO:teuthology.orchestra.run.smithi078.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-15T03:38:04.002 INFO:teuthology.orchestra.run.smithi078.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 366 kB/s | 18 kB 00:00 2024-09-15T03:38:04.027 INFO:teuthology.orchestra.run.smithi078.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-15T03:38:04.028 INFO:teuthology.orchestra.run.smithi105.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-15T03:38:04.053 INFO:teuthology.orchestra.run.smithi078.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-15T03:38:04.078 INFO:teuthology.orchestra.run.smithi078.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-15T03:38:04.103 INFO:teuthology.orchestra.run.smithi078.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-15T03:38:04.128 INFO:teuthology.orchestra.run.smithi078.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:38:04.154 INFO:teuthology.orchestra.run.smithi078.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-15T03:38:04.179 INFO:teuthology.orchestra.run.smithi078.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-15T03:38:04.186 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:04.187 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:04.196 INFO:teuthology.orchestra.run.smithi078.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T03:38:04.211 INFO:teuthology.orchestra.run.smithi105.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.7 MB/s | 1.4 MB 00:00 2024-09-15T03:38:04.221 INFO:teuthology.orchestra.run.smithi078.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-15T03:38:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-15T03:38:04.262 INFO:teuthology.orchestra.run.smithi105.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.9 MB/s | 2.1 MB 00:00 2024-09-15T03:38:04.263 INFO:teuthology.orchestra.run.smithi078.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-15T03:38:04.289 INFO:teuthology.orchestra.run.smithi078.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-15T03:38:04.314 INFO:teuthology.orchestra.run.smithi078.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-15T03:38:04.318 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:04.318 INFO:teuthology.orchestra.run.smithi078.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-15T03:38:04.387 INFO:teuthology.orchestra.run.smithi105.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-15T03:38:04.412 INFO:teuthology.orchestra.run.smithi105.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 1.6 MB/s | 240 kB 00:00 2024-09-15T03:38:04.446 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:04.484 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:04.484 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:04.729 INFO:teuthology.orchestra.run.smithi105.stdout:(6/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-15T03:38:04.876 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:04.876 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:04.938 INFO:teuthology.orchestra.run.smithi105.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 460 kB/s | 253 kB 00:00 2024-09-15T03:38:05.046 INFO:teuthology.orchestra.run.smithi105.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-15T03:38:05.163 INFO:teuthology.orchestra.run.smithi105.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 1.0 MB/s | 125 kB 00:00 2024-09-15T03:38:05.197 INFO:teuthology.orchestra.run.smithi105.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 228 kB/s | 106 kB 00:00 2024-09-15T03:38:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:38:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:38:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:38:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:38:05.280 INFO:teuthology.orchestra.run.smithi105.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-15T03:38:05.314 INFO:teuthology.orchestra.run.smithi105.stdout:(12/34): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2024-09-15T03:38:05.414 INFO:teuthology.orchestra.run.smithi105.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-15T03:38:05.448 INFO:teuthology.orchestra.run.smithi105.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 686 kB/s | 172 kB 00:00 2024-09-15T03:38:05.473 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:05.474 INFO:teuthology.orchestra.run.smithi105.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 866 kB/s | 51 kB 00:00 2024-09-15T03:38:05.499 INFO:teuthology.orchestra.run.smithi105.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 419 kB/s | 77 kB 00:00 2024-09-15T03:38:05.539 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:38:05.541 INFO:teuthology.orchestra.run.smithi105.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 649 kB/s | 43 kB 00:00 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:38:05.561 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:05.562 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:05.566 INFO:teuthology.orchestra.run.smithi105.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 354 kB/s | 42 kB 00:00 2024-09-15T03:38:05.580 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:38:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:38:05.692 INFO:teuthology.orchestra.run.smithi105.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-09-15T03:38:05.717 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:38:05.733 INFO:teuthology.orchestra.run.smithi105.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2024-09-15T03:38:05.759 INFO:teuthology.orchestra.run.smithi105.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 272 kB/s | 18 kB 00:00 2024-09-15T03:38:05.784 INFO:teuthology.orchestra.run.smithi105.stdout:(22/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-15T03:38:05.804 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:38:05.809 INFO:teuthology.orchestra.run.smithi105.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-15T03:38:05.834 INFO:teuthology.orchestra.run.smithi105.stdout:(24/34): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-09-15T03:38:05.850 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:38:05.860 INFO:teuthology.orchestra.run.smithi105.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-15T03:38:05.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-09-15T03:38:05.888 INFO:teuthology.orchestra.run.smithi105.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 588 kB/s | 46 kB 00:00 2024-09-15T03:38:05.892 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:38:05.914 INFO:teuthology.orchestra.run.smithi105.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-15T03:38:05.922 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:38:05.939 INFO:teuthology.orchestra.run.smithi105.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-15T03:38:05.957 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:38:05.972 INFO:teuthology.orchestra.run.smithi105.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-15T03:38:05.990 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:38:05.997 INFO:teuthology.orchestra.run.smithi105.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-09-15T03:38:06.031 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:38:06.031 INFO:teuthology.orchestra.run.smithi105.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-15T03:38:06.057 INFO:teuthology.orchestra.run.smithi105.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-15T03:38:06.075 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:38:06.082 INFO:teuthology.orchestra.run.smithi105.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-15T03:38:06.108 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:38:06.116 INFO:teuthology.orchestra.run.smithi105.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.5 MB/s | 427 kB 00:00 2024-09-15T03:38:06.120 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:06.121 INFO:teuthology.orchestra.run.smithi105.stdout:Total 9.3 MB/s | 27 MB 00:02 2024-09-15T03:38:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:06.166 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:38:06.214 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:38:06.252 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:38:06.289 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:06.289 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:06.290 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:38:06.307 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:38:06.400 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:38:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:38:06.459 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:43 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:38:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:38:06.534 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:38:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:38:06.588 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:38:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:38:06.612 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:06.614 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:38:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:38:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:38:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout:Install 34 Packages 2024-09-15T03:38:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:38:06.619 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 27 M 2024-09-15T03:38:06.619 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 96 M 2024-09-15T03:38:06.619 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:06.625 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:38:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:38:06.684 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:38:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:38:06.694 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:06.695 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:06.726 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:38:06.748 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:38:06.761 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:38:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:38:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:06.918 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:38:06.952 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:38:07.013 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:38:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:38:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:38:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:38:07.189 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:38:07.193 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:38:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:38:07.237 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:38:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:38:07.348 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:38:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:38:07.497 INFO:teuthology.orchestra.run.smithi064.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-15T03:38:07.515 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:38:07.534 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:38:07.547 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:07.575 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:07.575 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:38:07.575 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:07.575 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:38:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:38:07.689 INFO:teuthology.orchestra.run.smithi064.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.5 MB/s | 2.1 MB 00:00 2024-09-15T03:38:07.723 INFO:teuthology.orchestra.run.smithi064.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-09-15T03:38:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:38:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:38:07.790 INFO:teuthology.orchestra.run.smithi064.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 3.5 MB/s | 240 kB 00:00 2024-09-15T03:38:07.898 INFO:teuthology.orchestra.run.smithi064.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 9.8 MB/s | 3.9 MB 00:00 2024-09-15T03:38:07.904 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:38:07.941 INFO:teuthology.orchestra.run.smithi064.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-15T03:38:07.951 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:07.982 INFO:teuthology.orchestra.run.smithi064.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T03:38:08.024 INFO:teuthology.orchestra.run.smithi064.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T03:38:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:38:08.052 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:38:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:38:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:08.101 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:38:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:08.109 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:08.109 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:38:08.109 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:08.109 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:08.191 INFO:teuthology.orchestra.run.smithi064.stdout:(9/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-15T03:38:08.192 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:38:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 650 kB/s | 125 kB 00:00 2024-09-15T03:38:08.250 INFO:teuthology.orchestra.run.smithi064.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 3.5 MB/s | 1.2 MB 00:00 2024-09-15T03:38:08.278 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:38:08.325 INFO:teuthology.orchestra.run.smithi064.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 465 kB/s | 35 kB 00:00 2024-09-15T03:38:08.328 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:38:08.350 INFO:teuthology.orchestra.run.smithi064.stdout:(13/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-15T03:38:08.363 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:38:08.376 INFO:teuthology.orchestra.run.smithi064.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-15T03:38:08.393 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:38:08.401 INFO:teuthology.orchestra.run.smithi064.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-15T03:38:08.426 INFO:teuthology.orchestra.run.smithi064.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 825 kB/s | 42 kB 00:00 2024-09-15T03:38:08.427 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:38:08.452 INFO:teuthology.orchestra.run.smithi064.stdout:(17/34): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-09-15T03:38:08.477 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:38:08.502 INFO:teuthology.orchestra.run.smithi064.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-15T03:38:08.517 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:38:08.527 INFO:teuthology.orchestra.run.smithi064.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 465 kB/s | 11 kB 00:00 2024-09-15T03:38:08.553 INFO:teuthology.orchestra.run.smithi064.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 704 kB/s | 18 kB 00:00 2024-09-15T03:38:08.561 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:38:08.578 INFO:teuthology.orchestra.run.smithi064.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-15T03:38:08.596 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:38:08.603 INFO:teuthology.orchestra.run.smithi064.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-15T03:38:08.629 INFO:teuthology.orchestra.run.smithi064.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-09-15T03:38:08.644 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:38:08.645 INFO:teuthology.orchestra.run.smithi064.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 469 kB/s | 20 kB 00:00 2024-09-15T03:38:08.672 INFO:teuthology.orchestra.run.smithi064.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-15T03:38:08.685 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:08.696 INFO:teuthology.orchestra.run.smithi064.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:38:08.722 INFO:teuthology.orchestra.run.smithi064.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-15T03:38:08.739 INFO:teuthology.orchestra.run.smithi064.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 390 kB/s | 16 kB 00:00 2024-09-15T03:38:08.755 INFO:teuthology.orchestra.run.smithi064.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T03:38:08.757 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:38:08.780 INFO:teuthology.orchestra.run.smithi064.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-15T03:38:08.806 INFO:teuthology.orchestra.run.smithi064.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-15T03:38:08.831 INFO:teuthology.orchestra.run.smithi064.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-15T03:38:08.848 INFO:teuthology.orchestra.run.smithi064.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-09-15T03:38:08.873 INFO:teuthology.orchestra.run.smithi064.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-15T03:38:08.877 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:08.877 INFO:teuthology.orchestra.run.smithi064.stdout:Total 12 MB/s | 27 MB 00:02 2024-09-15T03:38:08.895 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:38:08.973 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:08.984 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:38:09.002 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:09.002 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:09.002 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:38:09.003 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:09.003 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:09.003 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:09.006 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:09.046 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:38:09.047 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:09.048 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:09.084 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:38:09.133 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:38:09.189 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:38:09.376 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:38:09.399 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:09.423 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:38:09.424 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:09.424 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:09.424 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:38:09.425 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:09.425 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:09.425 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:09.464 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:09.464 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:09.469 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:09.499 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:09.499 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:38:09.499 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:09.500 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:09.509 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:38:09.600 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:38:09.633 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:09.639 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:38:09.659 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:09.659 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:09.659 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:38:09.659 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:09.659 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:09.659 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:09.732 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:09.779 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:38:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:09.912 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:09.912 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:09.913 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:38:09.913 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:09.913 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:38:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:09.983 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:09.983 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:09.983 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:38:09.983 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:09.984 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:09.984 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:10.138 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:10.138 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:10.138 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:38:10.138 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:10.138 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:10.138 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:10.202 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:10.724 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:38:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:38:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:38:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:38:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:38:11.167 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:38:11.202 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:38:11.235 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:38:11.277 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:38:11.309 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:11.314 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:38:11.336 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:11.336 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:11.336 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:38:11.336 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:11.337 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:11.337 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:11.348 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:38:11.531 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:11.531 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:38:11.531 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:38:11.531 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:38:11.531 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:38:11.532 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:38:11.533 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:38:11.534 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:38:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:38:11.718 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:11.740 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:38:11.741 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:11.742 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:11.742 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:38:11.742 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:11.742 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:11.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:11.791 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:38:11.829 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:38:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:38:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:38:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:38:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:38:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:38:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:38:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:38:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:38:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:38:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:38:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:38:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:38:11.944 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:38:11.967 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:11.967 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:11.967 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:38:11.968 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:11.968 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:11.968 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:12.032 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:38:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:38:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:38:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:38:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:38:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:38:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:38:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:38:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:12.606 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:38:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:12.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:38:12.756 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:12.756 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:12.756 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.756 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:38:12.757 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:38:12.758 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:12.759 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:38:12.760 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:12.761 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:38:12.761 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:12.761 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:38:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:12.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-base 2024-09-15T03:38:13.128 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-base 2024-09-15T03:38:13.526 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:13.620 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:13.664 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:13.665 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:13.665 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:13.699 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:49 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:13.737 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install cephadm 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:38:13.745 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:38:13.746 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:38:13.747 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:38:13.748 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:38:13.787 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:13.830 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:13.830 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:13.830 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:13.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install cephadm 2024-09-15T03:38:14.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:14.189 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:38:14.189 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:14.189 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:38:14.189 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:14.189 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:38:14.189 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:14.325 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:14.456 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:50 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:14.462 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:14.462 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:38:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:14.464 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 83 k 2024-09-15T03:38:14.464 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 375 k 2024-09-15T03:38:14.464 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:14.577 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:14.587 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:38:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:14.589 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 83 k 2024-09-15T03:38:14.589 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 375 k 2024-09-15T03:38:14.589 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:14.602 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:38:14.603 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:14.603 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:38:14.603 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:14.603 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:38:14.603 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:14.649 INFO:teuthology.orchestra.run.smithi005.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 445 kB/s | 83 kB 00:00 2024-09-15T03:38:14.650 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:14.650 INFO:teuthology.orchestra.run.smithi005.stdout:Total 442 kB/s | 83 kB 00:00 2024-09-15T03:38:14.650 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:14.656 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:14.656 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:14.658 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:14.658 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:14.670 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:38:14.671 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:38:14.672 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:14.673 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:14.737 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:14.796 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 400 kB/s | 83 kB 00:00 2024-09-15T03:38:14.796 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:14.797 INFO:teuthology.orchestra.run.smithi078.stdout:Total 396 kB/s | 83 kB 00:00 2024-09-15T03:38:14.797 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:14.804 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:14.844 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:38:14.845 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:14.845 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:38:14.845 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:14.845 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:38:14.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:14.878 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:15.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2024-09-15T03:38:15.204 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.299 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.312 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.399 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.635 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:15.673 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.725 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:15.766 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:15.767 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:38:15.767 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:15.784 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.835 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2024-09-15T03:38:15.956 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:15.956 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:15.956 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:15.956 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:15.956 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:15.956 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:16.019 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:16.020 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:16.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:38:16.251 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:38:16.403 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:16.541 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:16.542 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:16.542 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:16.542 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:16.542 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 83 k 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 375 k 2024-09-15T03:38:16.543 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:16.577 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:38:16.577 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:38:16.577 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:38:16.577 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:38:16.577 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:38:16.579 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:38:16.580 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:38:16.580 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:38:16.580 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:38:16.580 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:38:16.581 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:38:16.581 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:38:16.737 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 427 kB/s | 83 kB 00:00 2024-09-15T03:38:16.737 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:16.738 INFO:teuthology.orchestra.run.smithi105.stdout:Total 423 kB/s | 83 kB 00:00 2024-09-15T03:38:16.738 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:16.744 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:16.744 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:16.746 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:16.746 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:16.762 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:16.822 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:16.826 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:16.900 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 137 k 2024-09-15T03:38:16.901 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 379 k 2024-09-15T03:38:16.902 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:16.958 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:16.958 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:16.958 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-09-15T03:38:16.958 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:16.958 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:16.958 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 137 k 2024-09-15T03:38:16.959 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 379 k 2024-09-15T03:38:16.960 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:17.091 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 727 kB/s | 137 kB 00:00 2024-09-15T03:38:17.092 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:17.092 INFO:teuthology.orchestra.run.smithi005.stdout:Total 721 kB/s | 137 kB 00:00 2024-09-15T03:38:17.093 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:17.100 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:17.100 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:17.160 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:17.161 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 685 kB/s | 137 kB 00:00 2024-09-15T03:38:17.161 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:17.161 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:17.162 INFO:teuthology.orchestra.run.smithi078.stdout:Total 679 kB/s | 137 kB 00:00 2024-09-15T03:38:17.162 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:17.169 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:17.169 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:17.225 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:17.225 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:17.285 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:17.321 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:17.365 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:17.383 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:17.384 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:17.410 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:17.410 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:17.411 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:38:17.411 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:17.448 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:17.473 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:17.473 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:17.473 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:38:17.473 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:17.475 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:38:17.476 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:38:17.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:38:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:38:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:38:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:38:17.478 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:17.478 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:17.777 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:17.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:18.248 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:38:18.335 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:18.336 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:18.336 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:18.336 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:18.336 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:18.336 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:18.398 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:18.462 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:18.462 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:18.490 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:18.534 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:18.535 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:38:18.535 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:18.554 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr 2024-09-15T03:38:18.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-09-15T03:38:18.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr 2024-09-15T03:38:18.819 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:18.955 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:18.955 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:18.955 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-09-15T03:38:18.955 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:38:18.956 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:18.957 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 137 k 2024-09-15T03:38:18.957 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 379 k 2024-09-15T03:38:18.957 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:19.141 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:19.168 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 652 kB/s | 137 kB 00:00 2024-09-15T03:38:19.169 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:19.169 INFO:teuthology.orchestra.run.smithi105.stdout:Total 647 kB/s | 137 kB 00:00 2024-09-15T03:38:19.169 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:19.176 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:19.176 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:19.201 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:19.233 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:19.233 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:19.235 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:19.280 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:19.282 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:19.282 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:19.287 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:19.339 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 83 k 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 375 k 2024-09-15T03:38:19.340 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:19.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:38:19.370 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:19.379 INFO:teuthology.orchestra.run.smithi078.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:19.420 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:19.421 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:19.421 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:19.436 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:19.461 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:19.461 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:19.461 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:38:19.461 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:19.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:38:19.527 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 443 kB/s | 83 kB 00:00 2024-09-15T03:38:19.527 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:19.528 INFO:teuthology.orchestra.run.smithi064.stdout:Total 439 kB/s | 83 kB 00:00 2024-09-15T03:38:19.528 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:19.533 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:19.533 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:19.535 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:19.536 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:19.959 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:20.060 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:20.098 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:20.098 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:38:20.099 INFO:teuthology.orchestra.run.smithi005.stdout:Installing weak dependencies: 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout:Install 7 Packages 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:20.100 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 3.7 M 2024-09-15T03:38:20.101 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 80 M 2024-09-15T03:38:20.101 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:20.193 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:20.193 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:38:20.194 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout:Install 7 Packages 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 3.7 M 2024-09-15T03:38:20.195 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 80 M 2024-09-15T03:38:20.196 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:20.313 INFO:teuthology.orchestra.run.smithi005.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 160 kB/s | 24 kB 00:00 2024-09-15T03:38:20.338 INFO:teuthology.orchestra.run.smithi005.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 82 kB/s | 15 kB 00:00 2024-09-15T03:38:20.351 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:20.351 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:20.352 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:20.352 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:20.352 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:20.352 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:20.380 INFO:teuthology.orchestra.run.smithi005.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:38:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:38:20.431 INFO:teuthology.orchestra.run.smithi005.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 445 kB/s | 41 kB 00:00 2024-09-15T03:38:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:38:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:20.507 INFO:teuthology.orchestra.run.smithi078.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 163 kB/s | 24 kB 00:00 2024-09-15T03:38:20.532 INFO:teuthology.orchestra.run.smithi078.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 83 kB/s | 15 kB 00:00 2024-09-15T03:38:20.574 INFO:teuthology.orchestra.run.smithi078.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:38:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.0 MB/s | 3.4 MB 00:00 2024-09-15T03:38:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Total 7.5 MB/s | 3.7 MB 00:00 2024-09-15T03:38:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:20.596 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2024-09-15T03:38:20.599 INFO:teuthology.orchestra.run.smithi078.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:38:20.625 INFO:teuthology.orchestra.run.smithi078.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-15T03:38:20.650 INFO:teuthology.orchestra.run.smithi078.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:38:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:20.750 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:20.750 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.2 MB/s | 3.4 MB 00:00 2024-09-15T03:38:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:Total 6.3 MB/s | 3.7 MB 00:00 2024-09-15T03:38:20.771 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:20.779 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:20.779 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:20.862 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:20.862 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:20.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:38:21.005 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:21.066 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:38:21.100 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:38:21.156 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:38:21.170 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:21.187 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:38:21.225 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:38:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:38:21.261 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:38:21.303 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:21.304 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:38:21.304 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:21.333 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:38:21.364 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:38:21.374 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:38:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:38:21.561 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:21.698 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:38:21.699 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:21.700 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 137 k 2024-09-15T03:38:21.700 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 379 k 2024-09-15T03:38:21.700 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:21.900 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 688 kB/s | 137 kB 00:00 2024-09-15T03:38:21.900 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:21.901 INFO:teuthology.orchestra.run.smithi064.stdout:Total 682 kB/s | 137 kB 00:00 2024-09-15T03:38:21.901 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:21.909 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:21.909 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:21.935 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:21.969 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:21.970 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:22.068 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:22.068 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-09-15T03:38:22.069 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout:Install 7 Packages 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.7 M 2024-09-15T03:38:22.070 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 80 M 2024-09-15T03:38:22.071 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:22.210 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:22.210 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:38:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:22.247 INFO:teuthology.orchestra.run.smithi105.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 167 kB/s | 24 kB 00:00 2024-09-15T03:38:22.273 INFO:teuthology.orchestra.run.smithi105.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 85 kB/s | 15 kB 00:00 2024-09-15T03:38:22.277 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:38:22.314 INFO:teuthology.orchestra.run.smithi105.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:38:22.329 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:22.340 INFO:teuthology.orchestra.run.smithi105.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:38:22.365 INFO:teuthology.orchestra.run.smithi105.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 445 kB/s | 41 kB 00:00 2024-09-15T03:38:22.390 INFO:teuthology.orchestra.run.smithi105.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:38:22.394 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:38:22.430 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:22.507 INFO:teuthology.orchestra.run.smithi105.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.3 MB/s | 3.4 MB 00:00 2024-09-15T03:38:22.510 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:22.511 INFO:teuthology.orchestra.run.smithi105.stdout:Total 8.3 MB/s | 3.7 MB 00:00 2024-09-15T03:38:22.511 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:22.519 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:22.519 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:22.611 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:22.611 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:22.962 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:23.033 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:38:23.067 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:38:23.121 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:38:23.154 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:38:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:38:23.158 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:23.158 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:23.159 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:23.159 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:23.183 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:38:23.352 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:23.352 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:38:23.353 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:38:23.353 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:38:23.353 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:38:23.353 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:38:23.353 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:38:23.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:38:23.432 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:38:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:23.637 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:38:23.727 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:23.728 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:23.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:38:23.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:38:23.997 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:24.090 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:24.135 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:24.136 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:38:24.136 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:24.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:38:24.203 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:24.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:38:24.434 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:02 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:24.504 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:24.575 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:24.576 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================= 2024-09-15T03:38:24.576 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:24.576 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================= 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:38:24.577 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================= 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout:Install 12 Packages 2024-09-15T03:38:24.578 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:24.579 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 38 M 2024-09-15T03:38:24.579 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 212 M 2024-09-15T03:38:24.579 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:24.638 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================= 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================= 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:38:24.639 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout:======================================================================================= 2024-09-15T03:38:24.640 INFO:teuthology.orchestra.run.smithi078.stdout:Install 12 Packages 2024-09-15T03:38:24.641 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:24.641 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 38 M 2024-09-15T03:38:24.641 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 212 M 2024-09-15T03:38:24.641 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:24.799 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:01 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:24.939 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:24.939 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:24.939 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:24.939 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:24.940 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:24.941 INFO:teuthology.orchestra.run.smithi064.stdout:Install 7 Packages 2024-09-15T03:38:24.941 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:24.941 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.7 M 2024-09-15T03:38:24.941 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 80 M 2024-09-15T03:38:24.941 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:25.220 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:25.220 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:38:25.220 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:38:25.220 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:38:25.220 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:38:25.220 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:38:25.221 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:38:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-15T03:38:25.333 INFO:teuthology.orchestra.run.smithi078.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 438 kB/s | 33 kB 00:00 2024-09-15T03:38:25.429 INFO:teuthology.orchestra.run.smithi064.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 155 kB/s | 24 kB 00:00 2024-09-15T03:38:25.454 INFO:teuthology.orchestra.run.smithi064.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 80 kB/s | 15 kB 00:00 2024-09-15T03:38:25.496 INFO:teuthology.orchestra.run.smithi064.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:38:25.517 INFO:teuthology.orchestra.run.smithi078.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-09-15T03:38:25.521 INFO:teuthology.orchestra.run.smithi064.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:38:25.523 INFO:teuthology.orchestra.run.smithi005.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 336 kB/s | 191 kB 00:00 2024-09-15T03:38:25.546 INFO:teuthology.orchestra.run.smithi064.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 442 kB/s | 41 kB 00:00 2024-09-15T03:38:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:38:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:25.549 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:38:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:38:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:38:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:38:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:25.550 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:25.551 INFO:teuthology.orchestra.run.smithi078.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 549 kB/s | 18 kB 00:00 2024-09-15T03:38:25.571 INFO:teuthology.orchestra.run.smithi064.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:38:25.592 INFO:teuthology.orchestra.run.smithi078.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 929 kB/s | 38 kB 00:00 2024-09-15T03:38:25.606 INFO:teuthology.orchestra.run.smithi005.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 394 kB/s | 33 kB 00:00 2024-09-15T03:38:25.626 INFO:teuthology.orchestra.run.smithi078.stdout:(6/12): libquadmath-11.5.0-2.el9.x86_64.rpm 229 kB/s | 191 kB 00:00 2024-09-15T03:38:25.689 INFO:teuthology.orchestra.run.smithi064.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.0 MB/s | 3.4 MB 00:00 2024-09-15T03:38:25.690 INFO:teuthology.orchestra.run.smithi005.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-15T03:38:25.692 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:25.693 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.9 MB/s | 3.7 MB 00:00 2024-09-15T03:38:25.693 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:25.702 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:25.702 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:25.727 INFO:teuthology.orchestra.run.smithi078.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 37 MB/s | 4.9 MB 00:00 2024-09-15T03:38:25.765 INFO:teuthology.orchestra.run.smithi005.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 243 kB/s | 18 kB 00:00 2024-09-15T03:38:25.790 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:25.791 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:25.792 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:38:25.824 INFO:teuthology.orchestra.run.smithi005.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-09-15T03:38:25.835 INFO:teuthology.orchestra.run.smithi078.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 47 MB/s | 5.1 MB 00:00 2024-09-15T03:38:25.849 INFO:teuthology.orchestra.run.smithi005.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 461 kB/s | 38 kB 00:00 2024-09-15T03:38:25.860 INFO:teuthology.orchestra.run.smithi078.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-15T03:38:25.886 INFO:teuthology.orchestra.run.smithi078.stdout:(10/12): libgfortran-11.5.0-2.el9.x86_64.rpm 732 kB/s | 801 kB 00:01 2024-09-15T03:38:25.911 INFO:teuthology.orchestra.run.smithi078.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 3.3 MB/s | 250 kB 00:00 2024-09-15T03:38:25.950 INFO:teuthology.orchestra.run.smithi005.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 39 MB/s | 4.9 MB 00:00 2024-09-15T03:38:26.042 INFO:teuthology.orchestra.run.smithi005.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-15T03:38:26.106 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:26.142 INFO:teuthology.orchestra.run.smithi005.stdout:(9/12): ceph-mgr-diskprediction-local-17.2.7-16 6.2 MB/s | 7.4 MB 00:01 2024-09-15T03:38:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:38:26.167 INFO:teuthology.orchestra.run.smithi005.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-15T03:38:26.200 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:38:26.211 INFO:teuthology.orchestra.run.smithi078.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-09-15T03:38:26.212 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:26.213 INFO:teuthology.orchestra.run.smithi078.stdout:Total 24 MB/s | 38 MB 00:01 2024-09-15T03:38:26.243 INFO:teuthology.orchestra.run.smithi005.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-15T03:38:26.247 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:38:26.277 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:38:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:38:26.355 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:03 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:26.488 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:38:26.489 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:26.490 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-09-15T03:38:26.491 INFO:teuthology.orchestra.run.smithi105.stdout:Install 12 Packages 2024-09-15T03:38:26.491 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:26.491 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 38 M 2024-09-15T03:38:26.491 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 212 M 2024-09-15T03:38:26.491 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:26.570 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:26.596 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:26.596 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:26.601 INFO:teuthology.orchestra.run.smithi005.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-15T03:38:26.602 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:26.602 INFO:teuthology.orchestra.run.smithi005.stdout:Total 19 MB/s | 38 MB 00:02 2024-09-15T03:38:26.824 INFO:teuthology.orchestra.run.smithi105.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 903 kB/s | 191 kB 00:00 2024-09-15T03:38:26.882 INFO:teuthology.orchestra.run.smithi105.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-15T03:38:26.972 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:26.998 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:26.998 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:27.074 INFO:teuthology.orchestra.run.smithi105.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-15T03:38:27.094 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:27.359 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:38:27.401 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:27.425 INFO:teuthology.orchestra.run.smithi105.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-15T03:38:27.531 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:27.532 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:27.591 INFO:teuthology.orchestra.run.smithi105.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 36 kB/s | 18 kB 00:00 2024-09-15T03:38:27.848 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:38:27.955 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:27.983 INFO:teuthology.orchestra.run.smithi105.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-15T03:38:28.008 INFO:teuthology.orchestra.run.smithi105.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 66 kB/s | 38 kB 00:00 2024-09-15T03:38:28.050 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:28.085 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:38:28.277 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:38:28.317 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:38:28.390 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:38:28.397 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:38:28.440 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:28.460 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:28.461 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:38:28.461 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:38:28.461 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:38:28.461 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:38:28.461 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:38:28.461 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:38:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:38:28.734 INFO:teuthology.orchestra.run.smithi105.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 326 kB/s | 245 kB 00:00 2024-09-15T03:38:28.741 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:38:28.741 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:28.741 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:28.741 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:28.742 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:28.788 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:38:28.816 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:38:28.817 INFO:teuthology.orchestra.run.smithi105.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.0 MB/s | 4.9 MB 00:01 2024-09-15T03:38:28.911 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:38:28.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:38:29.062 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:38:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:38:29.243 INFO:teuthology.orchestra.run.smithi105.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-09-15T03:38:29.551 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:06 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:29.585 INFO:teuthology.orchestra.run.smithi105.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 295 kB/s | 250 kB 00:00 2024-09-15T03:38:29.594 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:38:29.690 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:38:29.691 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:38:29.692 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 212 M 2024-09-15T03:38:29.693 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:29.709 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:38:30.055 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-15T03:38:30.163 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-09-15T03:38:30.189 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-15T03:38:30.297 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-09-15T03:38:30.381 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 462 kB/s | 38 kB 00:00 2024-09-15T03:38:30.410 INFO:teuthology.orchestra.run.smithi105.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-15T03:38:30.412 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:30.412 INFO:teuthology.orchestra.run.smithi105.stdout:Total 9.7 MB/s | 38 MB 00:03 2024-09-15T03:38:30.416 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:30.465 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 12 MB/s | 7.4 MB 00:00 2024-09-15T03:38:30.680 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-15T03:38:30.767 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:30.794 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:30.795 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:31.041 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:31.132 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-09-15T03:38:31.274 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-15T03:38:31.290 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:31.290 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:38:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:32.055 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:32.130 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:38:32.135 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:38:32.158 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:32.181 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:32.382 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.5 MB/s | 4.9 MB 00:02 2024-09-15T03:38:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:38:32.449 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.9 MB/s | 5.1 MB 00:01 2024-09-15T03:38:32.611 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:38:32.641 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:38:32.726 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:38:33.373 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:38:33.497 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:38:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:38:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:38:34.007 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-09-15T03:38:34.008 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:34.009 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.9 MB/s | 38 MB 00:04 2024-09-15T03:38:34.376 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:38:34.387 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:34.388 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:34.403 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:34.403 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:38:34.661 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:38:34.662 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:38:34.662 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:38:34.662 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:38:34.662 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:34.662 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:38:34.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:38:34.748 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:34.931 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:34.932 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:38:35.120 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:35.121 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:35.315 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:11 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:35.447 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:35.448 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:35.448 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:38:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2024-09-15T03:38:35.450 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:35.451 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 2.2 M 2024-09-15T03:38:35.451 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 26 M 2024-09-15T03:38:35.451 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:35.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:38:35.718 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:38:35.785 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:38:35.808 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:35.838 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:35.951 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:38:35.999 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 284 kB/s | 26 kB 00:00 2024-09-15T03:38:36.024 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 166 kB/s | 19 kB 00:00 2024-09-15T03:38:36.083 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 273 kB/s | 47 kB 00:00 2024-09-15T03:38:36.104 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:14 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:36.125 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-15T03:38:36.150 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-15T03:38:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:38:36.192 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 800 kB/s | 54 kB 00:00 2024-09-15T03:38:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:38:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 279 kB 00:00 2024-09-15T03:38:36.244 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:36.245 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:36.245 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:36.245 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:36.245 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:38:36.246 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout:Install 13 Packages 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:36.247 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 2.2 M 2024-09-15T03:38:36.248 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 26 M 2024-09-15T03:38:36.248 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:36.251 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 319 kB/s | 32 kB 00:00 2024-09-15T03:38:36.285 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 156 kB/s | 14 kB 00:00 2024-09-15T03:38:36.302 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:38:36.310 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-15T03:38:36.344 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-15T03:38:36.361 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.6 MB/s | 220 kB 00:00 2024-09-15T03:38:36.394 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-15T03:38:36.397 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:36.397 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-15T03:38:36.435 INFO:teuthology.orchestra.run.smithi005.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 170 kB/s | 19 kB 00:00 2024-09-15T03:38:36.461 INFO:teuthology.orchestra.run.smithi005.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 26 kB 00:00 2024-09-15T03:38:36.465 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:36.481 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:36.481 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:36.486 INFO:teuthology.orchestra.run.smithi005.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 291 kB/s | 47 kB 00:00 2024-09-15T03:38:36.567 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:36.568 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:36.569 INFO:teuthology.orchestra.run.smithi005.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-15T03:38:36.603 INFO:teuthology.orchestra.run.smithi005.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-15T03:38:36.628 INFO:teuthology.orchestra.run.smithi005.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-15T03:38:36.671 INFO:teuthology.orchestra.run.smithi005.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-15T03:38:36.696 INFO:teuthology.orchestra.run.smithi005.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-15T03:38:36.721 INFO:teuthology.orchestra.run.smithi005.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-15T03:38:36.746 INFO:teuthology.orchestra.run.smithi005.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-15T03:38:36.780 INFO:teuthology.orchestra.run.smithi005.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.3 MB/s | 90 kB 00:00 2024-09-15T03:38:36.813 INFO:teuthology.orchestra.run.smithi005.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-09-15T03:38:36.847 INFO:teuthology.orchestra.run.smithi005.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-15T03:38:36.850 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:36.850 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.7 MB/s | 2.2 MB 00:00 2024-09-15T03:38:36.916 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:36.932 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:36.932 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:38:37.028 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:37.028 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:37.107 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:38:37.241 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:38:37.286 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:38:37.341 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:38:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:38:37.412 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:38:37.514 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:38:37.548 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:37.624 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:38:37.680 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:38:37.727 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:38:37.784 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:38:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:38:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:38:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:38:38.053 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:38:38.159 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:38:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:38:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:38:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:38:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:38:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:38:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:38:38.395 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:38:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:38.453 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:38:38.479 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:38:38.524 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:38:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:38.582 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:38:38.583 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:38.584 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:38.584 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:38:38.584 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:38.584 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:38.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:38:38.981 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:38:39.033 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:38:39.067 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:38:39.113 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:38:39.135 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:38:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:38:39.518 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:38:39.549 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:39.550 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:16 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:39.682 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:39.683 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:39.683 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:39.683 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:39.683 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:38:39.684 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2024-09-15T03:38:39.685 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:39.686 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.2 M 2024-09-15T03:38:39.686 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-09-15T03:38:39.686 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:39.877 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 163 kB/s | 19 kB 00:00 2024-09-15T03:38:39.893 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 191 kB/s | 26 kB 00:00 2024-09-15T03:38:39.910 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 314 kB/s | 47 kB 00:00 2024-09-15T03:38:39.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:38:40.017 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-15T03:38:40.036 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-15T03:38:40.038 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:40.038 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:38:40.038 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:38:40.038 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:38:40.038 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:38:40.039 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:38:40.078 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 799 kB/s | 54 kB 00:00 2024-09-15T03:38:40.111 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-09-15T03:38:40.137 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-15T03:38:40.162 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-15T03:38:40.195 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-15T03:38:40.229 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-15T03:38:40.254 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 640 kB/s | 59 kB 00:00 2024-09-15T03:38:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-15T03:38:40.291 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:40.291 INFO:teuthology.orchestra.run.smithi105.stdout:Total 3.7 MB/s | 2.2 MB 00:00 2024-09-15T03:38:40.351 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:40.366 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:40.366 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:38:40.384 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:40.385 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:40.455 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:40.455 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:40.558 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:16 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:40.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:38:40.690 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:40.691 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:40.691 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:40.691 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:40.691 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout:Install 6 Packages 2024-09-15T03:38:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:40.693 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 7.0 M 2024-09-15T03:38:40.693 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 33 M 2024-09-15T03:38:40.693 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:40.954 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:41.121 INFO:teuthology.orchestra.run.smithi078.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-09-15T03:38:41.135 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:38:41.171 INFO:teuthology.orchestra.run.smithi078.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T03:38:41.196 INFO:teuthology.orchestra.run.smithi078.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-15T03:38:41.214 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:38:41.217 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:19 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:41.222 INFO:teuthology.orchestra.run.smithi078.stdout:(4/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 559 kB/s | 124 kB 00:00 2024-09-15T03:38:41.247 INFO:teuthology.orchestra.run.smithi078.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-15T03:38:41.270 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:38:41.307 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:38:41.348 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:38:41.357 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:41.358 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:41.358 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:41.358 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout:Install 6 Packages 2024-09-15T03:38:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:41.360 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 7.0 M 2024-09-15T03:38:41.360 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 33 M 2024-09-15T03:38:41.360 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:41.364 INFO:teuthology.orchestra.run.smithi078.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-15T03:38:41.366 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:41.366 INFO:teuthology.orchestra.run.smithi078.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-15T03:38:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:41.448 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:41.448 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:41.450 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:38:41.545 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:41.546 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:41.552 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:38:41.616 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:38:41.853 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:41.892 INFO:teuthology.orchestra.run.smithi005.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-15T03:38:41.942 INFO:teuthology.orchestra.run.smithi005.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T03:38:41.967 INFO:teuthology.orchestra.run.smithi005.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 577 kB/s | 124 kB 00:00 2024-09-15T03:38:41.993 INFO:teuthology.orchestra.run.smithi005.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T03:38:42.022 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:38:42.034 INFO:teuthology.orchestra.run.smithi005.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:38:42.044 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:38:42.045 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:42.045 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:38:42.074 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:38:42.118 INFO:teuthology.orchestra.run.smithi005.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-15T03:38:42.121 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:42.121 INFO:teuthology.orchestra.run.smithi005.stdout:Total 9.2 MB/s | 7.0 MB 00:00 2024-09-15T03:38:42.195 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:42.207 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:42.208 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:42.312 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:42.313 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:38:42.381 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:38:42.414 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:38:42.428 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:42.429 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:42.429 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:38:42.429 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:42.429 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:42.460 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:38:42.484 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:42.597 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:38:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:42.686 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:38:42.774 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:38:42.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:38:42.803 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:38:42.888 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:38:43.254 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:38:43.255 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:38:43.386 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:20 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:43.442 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:38:43.528 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:38:43.529 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:43.530 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:43.531 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:43.531 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-09-15T03:38:43.531 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:43.531 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-09-15T03:38:43.531 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-09-15T03:38:43.531 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:43.543 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:38:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:38:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:38:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:38:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:38:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:38:43.609 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:38:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:38:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:43.614 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:43.641 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:43.800 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 144 kB/s | 19 kB 00:00 2024-09-15T03:38:43.825 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 162 kB/s | 26 kB 00:00 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:38:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:43.851 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:43.851 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 259 kB/s | 47 kB 00:00 2024-09-15T03:38:43.878 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:38:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-15T03:38:43.959 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-15T03:38:44.001 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 804 kB/s | 54 kB 00:00 2024-09-15T03:38:44.043 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 387 kB/s | 32 kB 00:00 2024-09-15T03:38:44.085 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-15T03:38:44.110 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 134 kB/s | 14 kB 00:00 2024-09-15T03:38:44.112 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-fuse 2024-09-15T03:38:44.152 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.1 MB/s | 59 kB 00:00 2024-09-15T03:38:44.186 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-15T03:38:44.211 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.6 MB/s | 90 kB 00:00 2024-09-15T03:38:44.261 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-15T03:38:44.265 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:44.265 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-15T03:38:44.331 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:44.347 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:44.347 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:44.406 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:44.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:38:44.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:38:44.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:38:44.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:38:44.406 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:38:44.441 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:44.443 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:44.445 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:44.574 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:38:44.575 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 7.0 M 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 33 M 2024-09-15T03:38:44.576 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:44.683 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:20 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:38:44.691 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:38:44.692 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:44.692 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:44.817 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:44.818 INFO:teuthology.orchestra.run.smithi078.stdout:Install 2 Packages 2024-09-15T03:38:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:44.819 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 893 k 2024-09-15T03:38:44.819 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 2.6 M 2024-09-15T03:38:44.819 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:44.944 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-fuse 2024-09-15T03:38:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:45.099 INFO:teuthology.orchestra.run.smithi078.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-15T03:38:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:38:45.173 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-15T03:38:45.199 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 526 kB/s | 124 kB 00:00 2024-09-15T03:38:45.205 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:38:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.5 MB/s | 813 kB 00:00 2024-09-15T03:38:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:Total 2.0 MB/s | 893 kB 00:00 2024-09-15T03:38:45.261 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:38:45.271 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:45.274 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.4 MB/s | 548 kB 00:00 2024-09-15T03:38:45.293 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:45.293 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:45.297 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:38:45.299 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 575 kB/s | 58 kB 00:00 2024-09-15T03:38:45.324 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-15T03:38:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:38:45.375 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:45.375 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:45.454 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:38:45.533 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-09-15T03:38:45.535 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:45.536 INFO:teuthology.orchestra.run.smithi105.stdout:Total 7.3 MB/s | 7.0 MB 00:00 2024-09-15T03:38:45.541 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:38:45.606 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:45.616 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:38:45.617 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:45.617 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:45.682 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:38:45.683 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout:Install 2 Packages 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 893 k 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 2.6 M 2024-09-15T03:38:45.684 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:38:45.716 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:45.716 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:45.954 INFO:teuthology.orchestra.run.smithi005.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 555 kB/s | 80 kB 00:00 2024-09-15T03:38:46.027 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:46.095 INFO:teuthology.orchestra.run.smithi005.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-15T03:38:46.096 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:46.096 INFO:teuthology.orchestra.run.smithi005.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-09-15T03:38:46.109 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:46.132 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:46.133 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:46.207 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:38:46.216 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:46.216 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:46.260 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:38:46.376 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:38:46.412 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:46.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:38:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:38:46.463 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:38:46.509 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:38:46.532 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:46.560 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:38:46.640 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:46.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:38:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:46.758 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:38:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:46.758 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:46.784 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:38:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:38:46.934 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:38:46.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:46.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-volume 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:38:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:38:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:38:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:38:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:38:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:38:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:38:47.423 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:47.423 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:38:47.574 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:38:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:47.699 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:47.711 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:47.711 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:47.711 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-15T03:38:47.711 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:47.711 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:47.711 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:38:47.712 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:47.712 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:47.712 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:47.712 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:38:47.712 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:47.712 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 270 k 2024-09-15T03:38:47.713 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 1.3 M 2024-09-15T03:38:47.713 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:47.714 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:38:47.714 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:47.714 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:47.714 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:38:47.714 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:47.714 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:47.776 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:47.776 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:38:47.776 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:38:47.776 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:38:47.776 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:38:47.777 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:38:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T03:38:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T03:38:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:47.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:38:47.946 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:47.946 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:47.970 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-volume 2024-09-15T03:38:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:38:48.054 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:38:48.055 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:38:48.055 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:48.055 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:48.189 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:48.271 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:48.271 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:48.271 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:38:48.271 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:48.319 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2024-09-15T03:38:48.531 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:25 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:48.561 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:48.670 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:48.671 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:48.671 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:48.671 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:48.671 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:48.672 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-09-15T03:38:48.673 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:48.673 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-09-15T03:38:48.673 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-09-15T03:38:48.673 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:48.700 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:48.700 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 270 k 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 1.3 M 2024-09-15T03:38:48.701 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:48.906 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:25 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:48.918 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-15T03:38:48.919 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:48.919 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-15T03:38:48.919 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:48.929 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:48.929 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:48.957 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:48.957 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:49.013 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-09-15T03:38:49.046 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:49.046 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:49.046 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-09-15T03:38:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:49.048 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 893 k 2024-09-15T03:38:49.048 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.6 M 2024-09-15T03:38:49.048 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:49.055 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 599 kB/s | 124 kB 00:00 2024-09-15T03:38:49.089 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-15T03:38:49.114 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-15T03:38:49.131 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 761 kB/s | 58 kB 00:00 2024-09-15T03:38:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:49.214 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-15T03:38:49.217 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:49.217 INFO:teuthology.orchestra.run.smithi064.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-15T03:38:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:49.242 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:49.243 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:49.245 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:49.261 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:49.262 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:49.262 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:38:49.262 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:49.285 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 588 kB/s | 80 kB 00:00 2024-09-15T03:38:49.290 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:49.303 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:49.303 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:49.407 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:49.408 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:49.444 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-15T03:38:49.444 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:49.445 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.2 MB/s | 893 kB 00:00 2024-09-15T03:38:49.456 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:49.478 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:49.478 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:49.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados-devel 2024-09-15T03:38:49.559 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:49.559 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:49.730 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:49.896 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:38:49.896 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:38:49.951 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:38:49.967 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:50.086 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:50.224 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 126 k 2024-09-15T03:38:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 448 k 2024-09-15T03:38:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:50.298 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:50.298 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 609 kB/s | 126 kB 00:00 2024-09-15T03:38:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:Total 603 kB/s | 126 kB 00:00 2024-09-15T03:38:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:50.440 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:50.440 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:38:50.503 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:50.503 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:50.562 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados-devel 2024-09-15T03:38:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:38:50.654 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:50.669 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:50.669 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:38:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:38:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:50.772 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:38:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:50.945 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:38:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:50.945 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:51.160 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:51.193 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:51.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-volume 2024-09-15T03:38:51.300 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:51.300 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:51.300 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-15T03:38:51.300 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:51.301 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 126 k 2024-09-15T03:38:51.302 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 448 k 2024-09-15T03:38:51.302 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:51.394 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:51.394 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:51.502 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:51.503 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:38:51.503 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:38:51.503 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:38:51.503 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:38:51.503 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:38:51.508 INFO:teuthology.orchestra.run.smithi005.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 613 kB/s | 126 kB 00:00 2024-09-15T03:38:51.508 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:51.509 INFO:teuthology.orchestra.run.smithi005.stdout:Total 608 kB/s | 126 kB 00:00 2024-09-15T03:38:51.509 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:51.515 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:51.515 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:51.581 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:51.582 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:51.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs2 2024-09-15T03:38:51.734 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:51.788 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:38:51.788 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:51.788 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:51.788 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:51.789 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:51.791 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:51.858 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:51.928 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:51.928 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 270 k 2024-09-15T03:38:51.929 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 1.3 M 2024-09-15T03:38:51.930 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:52.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-09-15T03:38:52.174 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-15T03:38:52.174 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:52.175 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-15T03:38:52.175 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:52.183 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:52.183 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:52.214 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:52.215 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:52.216 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:52.306 INFO:teuthology.orchestra.run.smithi078.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:52.351 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:52.352 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:52.352 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:52.376 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:52.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install libcephfs-devel 2024-09-15T03:38:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:52.517 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:52.517 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:38:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:52.627 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:52.627 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:52.654 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:52.795 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:38:52.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:52.797 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:52.797 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:52.797 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-09-15T03:38:52.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:52.797 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 893 k 2024-09-15T03:38:52.797 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.6 M 2024-09-15T03:38:52.798 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:52.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs2 2024-09-15T03:38:52.975 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:38:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 25 k 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 110 k 2024-09-15T03:38:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:53.232 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 513 kB/s | 80 kB 00:00 2024-09-15T03:38:53.387 INFO:teuthology.orchestra.run.smithi078.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 89 kB/s | 25 kB 00:00 2024-09-15T03:38:53.388 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:53.388 INFO:teuthology.orchestra.run.smithi078.stdout:Total 89 kB/s | 25 kB 00:00 2024-09-15T03:38:53.388 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:53.391 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:53.391 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:53.393 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:53.393 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:53.460 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:38:53.542 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:53.542 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:53.542 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:53.542 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:53.543 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:53.554 INFO:teuthology.orchestra.run.smithi005.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:53.599 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:53.601 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:53.601 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:53.624 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 1.5 MB/s | 813 kB 00:00 2024-09-15T03:38:53.625 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:53.625 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.1 MB/s | 893 kB 00:00 2024-09-15T03:38:53.638 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:53.663 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:53.663 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:53.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs-devel 2024-09-15T03:38:53.749 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:53.749 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:53.828 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2024-09-15T03:38:53.938 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:38:54.136 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:54.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:54.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:54.205 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:38:54.205 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:54.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:54.205 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:54.277 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:54.378 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librados2 2024-09-15T03:38:54.405 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:54.416 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:38:54.417 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:38:54.418 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:38:54.418 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:54.418 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 25 k 2024-09-15T03:38:54.418 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 110 k 2024-09-15T03:38:54.418 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:38:54.543 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:54.544 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:54.544 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-09-15T03:38:54.544 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 126 k 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 448 k 2024-09-15T03:38:54.545 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:54.746 INFO:teuthology.orchestra.run.smithi005.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 76 kB/s | 25 kB 00:00 2024-09-15T03:38:54.747 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:54.747 INFO:teuthology.orchestra.run.smithi005.stdout:Total 76 kB/s | 25 kB 00:00 2024-09-15T03:38:54.747 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:38:54.751 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:38:54.751 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:38:54.752 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:38:54.752 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:38:54.843 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:38:54.860 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:54.860 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:38:54.942 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:30 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:54.962 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:54.962 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 303 kB/s | 126 kB 00:00 2024-09-15T03:38:54.963 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:54.963 INFO:teuthology.orchestra.run.smithi105.stdout:Total 301 kB/s | 126 kB 00:00 2024-09-15T03:38:54.963 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:54.969 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:54.969 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:55.031 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:55.032 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:55.032 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:55.074 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:55.075 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:55.075 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:55.143 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:38:55.143 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:55.143 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:55.143 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:38:55.143 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:55.143 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:55.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install librbd1 2024-09-15T03:38:55.184 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:55.292 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:55.375 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:55.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-09-15T03:38:55.623 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:55.623 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:55.623 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:38:55.623 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:55.623 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:38:55.624 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:55.709 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:55.717 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:55.798 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:55.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados2 2024-09-15T03:38:55.841 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:55.842 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:55.843 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:55.903 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rados 2024-09-15T03:38:55.985 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:55.986 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:55.986 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:55.986 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:55.986 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:55.986 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:55.996 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:56.135 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-15T03:38:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:38:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 270 k 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.3 M 2024-09-15T03:38:56.137 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:56.249 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2024-09-15T03:38:56.422 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:56.461 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:56.517 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:56.537 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 673 kB/s | 270 kB 00:00 2024-09-15T03:38:56.538 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:56.538 INFO:teuthology.orchestra.run.smithi064.stdout:Total 671 kB/s | 270 kB 00:00 2024-09-15T03:38:56.538 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:56.548 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:56.548 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:56.551 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:56.562 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:56.563 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:56.563 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:56.581 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:56.582 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:56.595 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:56.595 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:56.646 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librbd1 2024-09-15T03:38:56.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rgw 2024-09-15T03:38:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:56.825 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:56.898 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:56.915 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:56.915 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:56.915 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:38:56.915 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:56.916 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:56.962 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:56.963 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:38:56.963 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:57.041 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2024-09-15T03:38:57.221 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:57.259 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:57.352 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:57.353 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:57.355 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:57.355 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:57.398 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:57.399 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:57.399 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:57.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-cephfs 2024-09-15T03:38:57.477 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rados 2024-09-15T03:38:57.611 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:57.745 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:57.745 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:38:57.746 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:57.747 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 25 k 2024-09-15T03:38:57.747 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 110 k 2024-09-15T03:38:57.747 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:38:57.908 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 155 kB/s | 25 kB 00:00 2024-09-15T03:38:57.908 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:57.908 INFO:teuthology.orchestra.run.smithi105.stdout:Total 154 kB/s | 25 kB 00:00 2024-09-15T03:38:57.909 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:38:57.912 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:38:57.912 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:38:57.913 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:38:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:38:57.945 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:57.945 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:57.946 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:38:57.946 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:57.946 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:57.946 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:38:57.984 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:38:58.068 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:58.075 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:58.113 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:58.118 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:58.118 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:58.119 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:58.162 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:58.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install python3-rbd 2024-09-15T03:38:58.207 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:58.208 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:58.208 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:58.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-09-15T03:38:58.294 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rgw 2024-09-15T03:38:58.484 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:58.725 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:58.725 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:58.725 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:38:58.725 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:58.725 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:38:58.725 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:58.742 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:58.833 INFO:teuthology.orchestra.run.smithi078.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:58.854 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:58.875 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:58.876 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-09-15T03:38:58.876 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:38:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:58.921 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2024-09-15T03:38:58.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-fuse 2024-09-15T03:38:58.978 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:58.992 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:38:58.993 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:38:58.994 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 126 k 2024-09-15T03:38:58.994 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 448 k 2024-09-15T03:38:58.994 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:38:59.023 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:59.024 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:59.024 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:59.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-cephfs 2024-09-15T03:38:59.201 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 610 kB/s | 126 kB 00:00 2024-09-15T03:38:59.202 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:59.202 INFO:teuthology.orchestra.run.smithi064.stdout:Total 606 kB/s | 126 kB 00:00 2024-09-15T03:38:59.202 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:38:59.208 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:38:59.209 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:38:59.275 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:38:59.275 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:38:59.428 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:38:59.497 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:38:59.508 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:38:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:59.604 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:59.629 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:38:59.630 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:38:59.631 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:38:59.631 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:38:59.631 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 82 k 2024-09-15T03:38:59.631 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 222 k 2024-09-15T03:38:59.631 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:38:59.646 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:38:59.647 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:38:59.647 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:38:59.701 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:38:59.719 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2024-09-15T03:38:59.793 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:59.824 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 423 kB/s | 82 kB 00:00 2024-09-15T03:38:59.824 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:59.825 INFO:teuthology.orchestra.run.smithi078.stdout:Total 420 kB/s | 82 kB 00:00 2024-09-15T03:38:59.825 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:38:59.832 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:38:59.832 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:38:59.838 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:38:59.839 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:38:59.839 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:38:59.893 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:38:59.894 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:38:59.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rbd 2024-09-15T03:38:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:00.047 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:39:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:00.236 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:39:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:00.237 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:00.289 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:00.379 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:00.423 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:00.424 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:39:00.424 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:00.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-09-15T03:39:00.505 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2024-09-15T03:39:00.513 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:00.558 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:39:00.650 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:00.696 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:39:00.697 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-15T03:39:00.697 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:39:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:00.773 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:39:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:00.773 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:39:00.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-fuse 2024-09-15T03:39:01.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-mirror 2024-09-15T03:39:01.044 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:01.073 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:01.140 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:01.161 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:01.185 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:01.203 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:01.205 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:39:01.206 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:01.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-09-15T03:39:01.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2024-09-15T03:39:01.372 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:39:01.511 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:39:01.511 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:01.512 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 82 k 2024-09-15T03:39:01.513 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 222 k 2024-09-15T03:39:01.513 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:39:01.599 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:39:01.708 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 419 kB/s | 82 kB 00:00 2024-09-15T03:39:01.708 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:01.708 INFO:teuthology.orchestra.run.smithi005.stdout:Total 416 kB/s | 82 kB 00:00 2024-09-15T03:39:01.709 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:39:01.716 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:39:01.716 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:39:01.735 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repo Size 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:39:01.736 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:39:01.737 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:39:01.737 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:01.737 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 2.9 M 2024-09-15T03:39:01.737 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 12 M 2024-09-15T03:39:01.737 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:39:01.784 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:39:01.784 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:39:01.842 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:01.933 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:01.936 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:39:01.976 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:01.977 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:39:01.977 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:01.987 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:01.988 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:39:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:01.989 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 25 k 2024-09-15T03:39:01.989 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 110 k 2024-09-15T03:39:01.989 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:39:02.050 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.052 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2024-09-15T03:39:02.086 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.4 MB/s | 2.9 MB 00:00 2024-09-15T03:39:02.086 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:02.087 INFO:teuthology.orchestra.run.smithi078.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-09-15T03:39:02.087 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:39:02.095 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:39:02.096 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:39:02.138 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 168 kB/s | 25 kB 00:00 2024-09-15T03:39:02.139 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:02.139 INFO:teuthology.orchestra.run.smithi064.stdout:Total 166 kB/s | 25 kB 00:00 2024-09-15T03:39:02.139 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:39:02.143 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:39:02.143 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:39:02.144 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:39:02.144 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:39:02.175 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:39:02.175 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:39:02.235 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:39:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.480 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.590 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:39:02.618 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:02.645 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.670 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:39:02.670 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:39:02.670 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:02.670 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:02.707 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:02.716 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.747 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.747 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:02.747 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:39:02.747 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:02.747 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:02.747 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:39:02.750 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:02.751 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:39:02.751 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:02.827 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2024-09-15T03:39:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:02.962 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:39:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:02.962 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:03.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-mirror 2024-09-15T03:39:03.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-09-15T03:39:03.386 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:03.475 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:03.517 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:03.518 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-09-15T03:39:03.518 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:03.596 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2024-09-15T03:39:03.608 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:39:03.744 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:03.744 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:03.744 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:39:03.744 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:03.745 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:03.745 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:39:03.747 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:39:03.748 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:03.749 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 2.9 M 2024-09-15T03:39:03.749 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 12 M 2024-09-15T03:39:03.749 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:39:03.779 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:03.873 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:03.918 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:03.919 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:03.919 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:04.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install rbd-nbd 2024-09-15T03:39:04.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-09-15T03:39:04.160 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:04.212 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 6.4 MB/s | 2.9 MB 00:00 2024-09-15T03:39:04.212 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:04.213 INFO:teuthology.orchestra.run.smithi005.stdout:Total 6.3 MB/s | 2.9 MB 00:00 2024-09-15T03:39:04.213 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:39:04.223 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:39:04.223 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:39:04.293 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:39:04.294 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:04.295 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 82 k 2024-09-15T03:39:04.295 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 222 k 2024-09-15T03:39:04.295 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:39:04.309 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:39:04.309 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:39:04.478 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 448 kB/s | 82 kB 00:00 2024-09-15T03:39:04.478 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:04.478 INFO:teuthology.orchestra.run.smithi105.stdout:Total 444 kB/s | 82 kB 00:00 2024-09-15T03:39:04.479 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:39:04.485 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:39:04.485 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:39:04.547 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:39:04.547 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:39:04.580 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:37:24 AM UTC. 2024-09-15T03:39:04.595 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:04.689 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:04.702 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:39:04.713 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-15T03:39:04.713 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout:Install 1 Package 2024-09-15T03:39:04.714 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:04.715 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 165 k 2024-09-15T03:39:04.715 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 477 k 2024-09-15T03:39:04.715 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-15T03:39:04.734 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:04.735 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:04.735 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:04.745 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:39:04.808 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:04.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-09-15T03:39:04.821 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:04.848 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:04.848 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:39:04.848 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:39:04.848 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:04.848 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:04.848 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:04.906 INFO:teuthology.orchestra.run.smithi078.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 865 kB/s | 165 kB 00:00 2024-09-15T03:39:04.908 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:04.908 INFO:teuthology.orchestra.run.smithi078.stdout:Total 859 kB/s | 165 kB 00:00 2024-09-15T03:39:04.908 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-15T03:39:04.914 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-15T03:39:04.914 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-15T03:39:04.977 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-15T03:39:04.977 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-15T03:39:05.139 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-15T03:39:05.190 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:05.244 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:05.405 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:05.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:05.443 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:05.443 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:39:05.443 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:05.443 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:05.443 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:05.499 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:05.544 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:05.545 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:05.545 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:05.696 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2024-09-15T03:39:05.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-09-15T03:39:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:05.956 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-15T03:39:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:05.956 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:05.956 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-15T03:39:05.990 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:05.990 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:05.990 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:39:05.990 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:05.990 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:05.990 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:39:06.219 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:06.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-nbd 2024-09-15T03:39:06.271 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:06.338 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:06.409 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:39:06.410 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:06.411 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.9 M 2024-09-15T03:39:06.411 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 12 M 2024-09-15T03:39:06.411 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:39:06.431 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:06.476 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:06.477 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:06.477 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:06.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-09-15T03:39:06.748 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-15T03:39:06.748 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:06.749 INFO:teuthology.orchestra.run.smithi105.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-15T03:39:06.749 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:39:06.757 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:39:06.757 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:39:06.835 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:37:22 AM UTC. 2024-09-15T03:39:06.837 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:39:06.837 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:39:06.973 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-15T03:39:06.974 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:06.975 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 165 k 2024-09-15T03:39:06.976 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 477 k 2024-09-15T03:39:06.976 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-15T03:39:07.141 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:07.184 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 793 kB/s | 165 kB 00:00 2024-09-15T03:39:07.185 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:07.185 INFO:teuthology.orchestra.run.smithi005.stdout:Total 787 kB/s | 165 kB 00:00 2024-09-15T03:39:07.185 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-15T03:39:07.193 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-15T03:39:07.193 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-15T03:39:07.234 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:07.254 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:39:07.259 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-15T03:39:07.259 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-15T03:39:07.279 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:07.280 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:07.280 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:07.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-09-15T03:39:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-15T03:39:07.440 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:07.465 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:07.466 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:39:07.466 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:39:07.466 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:07.466 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:07.466 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:07.526 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:07.936 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:07.968 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:08.029 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:39:08.077 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:08.078 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-15T03:39:08.079 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:08.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-09-15T03:39:08.230 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:08.230 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:08.231 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-15T03:39:08.231 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:08.231 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:08.231 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-15T03:39:08.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:08.447 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:08.448 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:39:08.448 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:08.448 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:08.448 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:08.492 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:08.716 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2024-09-15T03:39:08.748 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:08.886 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:39:08.887 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:08.888 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:39:08.888 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:08.888 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 82 k 2024-09-15T03:39:08.888 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 222 k 2024-09-15T03:39:08.888 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:39:09.191 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 270 kB/s | 82 kB 00:00 2024-09-15T03:39:09.192 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:09.193 INFO:teuthology.orchestra.run.smithi064.stdout:Total 269 kB/s | 82 kB 00:00 2024-09-15T03:39:09.193 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:39:09.200 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:39:09.200 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:39:09.267 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:39:09.267 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:39:09.297 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:09.420 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:39:09.433 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-09-15T03:39:09.434 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-09-15T03:39:09.435 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-09-15T03:39:09.435 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:09.435 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 165 k 2024-09-15T03:39:09.435 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 477 k 2024-09-15T03:39:09.435 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-09-15T03:39:09.525 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:09.643 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 795 kB/s | 165 kB 00:00 2024-09-15T03:39:09.644 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:09.644 INFO:teuthology.orchestra.run.smithi105.stdout:Total 788 kB/s | 165 kB 00:00 2024-09-15T03:39:09.644 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-09-15T03:39:09.651 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-09-15T03:39:09.651 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-09-15T03:39:09.714 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-09-15T03:39:09.715 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-09-15T03:39:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-09-15T03:39:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:09.981 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:10.184 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:10.184 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:10.184 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:39:10.184 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:10.184 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:10.184 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:10.389 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:10.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-09-15T03:39:10.657 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:10.657 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:10.657 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-09-15T03:39:10.657 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:10.657 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:10.657 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-09-15T03:39:10.912 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:11.046 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:11.186 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:11.187 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:11.187 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-15T03:39:11.187 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:11.187 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.9 M 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 12 M 2024-09-15T03:39:11.188 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:39:11.528 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.6 MB/s | 2.9 MB 00:00 2024-09-15T03:39:11.529 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:11.529 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-09-15T03:39:11.529 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:39:11.539 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:39:11.539 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:39:11.623 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:39:11.623 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:39:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:39:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:12.146 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:39:12.147 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:39:12.147 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:12.147 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:39:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:13.318 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:39:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:13.318 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:13.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-09-15T03:39:14.208 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:51 ago on Sun 15 Sep 2024 03:37:23 AM UTC. 2024-09-15T03:39:14.351 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-15T03:39:14.351 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-15T03:39:14.352 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:14.353 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 165 k 2024-09-15T03:39:14.353 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 477 k 2024-09-15T03:39:14.353 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-15T03:39:14.533 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 919 kB/s | 165 kB 00:00 2024-09-15T03:39:14.534 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:39:14.534 INFO:teuthology.orchestra.run.smithi064.stdout:Total 910 kB/s | 165 kB 00:00 2024-09-15T03:39:14.534 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-15T03:39:14.543 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-15T03:39:14.543 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-15T03:39:14.610 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-15T03:39:14.610 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-15T03:39:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-15T03:39:14.867 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:15.302 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:15.585 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:39:15.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:15.585 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-15T03:39:15.585 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:39:15.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:15.586 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-15T03:39:15.839 DEBUG:teuthology.parallel:result is None 2024-09-15T03:39:15.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:39:16.002 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:39:16.028 INFO:teuthology.orchestra.run.smithi005.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.029 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.029 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:39:16.031 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:39:16.183 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:39:16.207 INFO:teuthology.orchestra.run.smithi064.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.207 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.207 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:39:16.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:39:16.360 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:39:16.385 INFO:teuthology.orchestra.run.smithi078.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.385 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.385 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:39:16.387 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:39:16.536 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:39:16.561 INFO:teuthology.orchestra.run.smithi105.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.561 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:39:16.561 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:39:16.563 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:39:16.563 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:16.563 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:39:16.595 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:16.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:39:16.628 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:16.628 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:39:16.658 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:16.658 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:39:16.688 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:39:16.688 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:16.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:39:16.719 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:39:16.790 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:16.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:39:16.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:39:16.890 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:16.890 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:39:16.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:39:16.986 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:16.987 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:39:17.016 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:39:17.084 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:39:17.084 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:17.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:39:17.110 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:39:17.177 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:17.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:39:17.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:39:17.278 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:17.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:39:17.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:39:17.368 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:17.368 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:39:17.393 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:39:17.460 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:39:17.460 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:17.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:39:17.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:39:17.553 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:17.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:39:17.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:39:17.655 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:17.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:39:17.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:39:17.745 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:17.745 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:39:17.770 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:39:17.835 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T03:39:17.930 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T03:39:17.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-09-15T03:39:17.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-09-15T03:39:17.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-09-15T03:39:17.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /var/log/ceph 2024-09-15T03:39:17.964 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T03:39:17.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:39:17.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:39:18.006 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:39:18.008 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:39:18.034 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T03:39:18.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:39:18.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:39:18.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:39:18.079 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:39:18.119 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T03:39:18.119 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T03:39:18.119 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905296', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNS5IfczFsQkpWcXWMM/nl3nQs9yaDUw9Fs/4qAacZ0wie8NcAueKBCPXPsUDrJXuI4BhhD26HnHonWJ17nqfoI=', 'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKW3rFnRTPQaoJAJ4HiOij90a231Rz749ywM96UK8r9PlhAt1ERSkD0lrlOa16C5/TPNF7Fu6zLPkbhRs/YJ94=', 'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBEjjg5rB/1inX87OniseCpFYKNd1SlbYYLJYN2JyrHK6uN7lmkXNKSCc7IpXymQoJXY/BMo7cq6WQeXCWMoJE=', 'smithi105.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS6yCoZepRmSPiKoN/amG2AnBigz51knwJTC7ZDatklNcKJpnfKUhTlzMhr5UWG7TijdNsDR/FHL6RDtN+Ogic='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T03:39:18.120 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:39:18.131 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:39:18.148 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:39:18.151 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:39:18.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:39:18.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:39:18.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:39:18.212 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:39:18.236 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:18.237 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:39:18.266 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:39:18.267 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-09-15T03:39:18.322 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:39:18.322 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:18.322 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-15T03:39:18.322 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:18.323 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:18.323 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-15 03:39:05.109018575 +0000 2024-09-15T03:39:18.323 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-15 03:36:40.500893375 +0000 2024-09-15T03:39:18.323 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-15 03:36:40.500893375 +0000 2024-09-15T03:39:18.323 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-15 03:36:40.500893375 +0000 2024-09-15T03:39:18.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:39:18.386 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-15T03:39:18.386 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-15T03:39:18.386 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000103811 s, 4.9 MB/s 2024-09-15T03:39:18.387 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:39:18.444 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-09-15T03:39:18.500 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:39:18.500 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:18.500 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T03:39:18.500 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:18.500 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:18.501 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-15 03:39:05.109018575 +0000 2024-09-15T03:39:18.501 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-15 03:36:40.829886759 +0000 2024-09-15T03:39:18.501 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-15 03:36:40.829886759 +0000 2024-09-15T03:39:18.501 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-15 03:36:40.829886759 +0000 2024-09-15T03:39:18.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:39:18.564 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-15T03:39:18.565 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-15T03:39:18.565 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.189e-05 s, 5.6 MB/s 2024-09-15T03:39:18.566 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:39:18.624 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-09-15T03:39:18.680 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:39:18.681 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:18.681 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-15T03:39:18.681 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:18.681 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:18.681 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-15 03:39:05.110018555 +0000 2024-09-15T03:39:18.681 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-15 03:36:41.147880365 +0000 2024-09-15T03:39:18.682 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-15 03:36:41.147880365 +0000 2024-09-15T03:39:18.682 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-15 03:36:41.147880365 +0000 2024-09-15T03:39:18.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:39:18.744 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-15T03:39:18.744 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-15T03:39:18.745 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 8.6356e-05 s, 5.9 MB/s 2024-09-15T03:39:18.746 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:39:18.803 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-09-15T03:39:18.860 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:39:18.860 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-15 03:39:05.110018555 +0000 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-15 03:36:41.455874175 +0000 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-15 03:36:41.455874175 +0000 2024-09-15T03:39:18.861 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-15 03:36:41.455874175 +0000 2024-09-15T03:39:18.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:39:18.924 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-15T03:39:18.924 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-15T03:39:18.924 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 8.0917e-05 s, 6.3 MB/s 2024-09-15T03:39:18.926 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:39:18.983 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:18.983 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:18.983 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:39:19.003 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:39:19.004 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-09-15T03:39:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:39:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:19.063 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T03:39:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-15 03:39:12.406550210 +0000 2024-09-15T03:39:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-15 03:36:40.941609792 +0000 2024-09-15T03:39:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-15 03:36:40.941609792 +0000 2024-09-15T03:39:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-15 03:36:40.941609792 +0000 2024-09-15T03:39:19.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:39:19.133 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-15T03:39:19.133 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-15T03:39:19.133 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000277197 s, 1.8 MB/s 2024-09-15T03:39:19.134 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:39:19.195 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-09-15T03:39:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:39:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:19.256 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T03:39:19.256 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:19.257 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:19.257 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-15 03:39:12.406550210 +0000 2024-09-15T03:39:19.257 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-15 03:36:41.273603448 +0000 2024-09-15T03:39:19.257 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-15 03:36:41.273603448 +0000 2024-09-15T03:39:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-15 03:36:41.273603448 +0000 2024-09-15T03:39:19.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:39:19.325 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-15T03:39:19.325 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-15T03:39:19.325 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000216581 s, 2.4 MB/s 2024-09-15T03:39:19.326 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:39:19.387 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-15 03:39:12.407550191 +0000 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-15 03:36:41.613596952 +0000 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-15 03:36:41.613596952 +0000 2024-09-15T03:39:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-15 03:36:41.613596952 +0000 2024-09-15T03:39:19.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:39:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-15T03:39:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-15T03:39:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000219976 s, 2.3 MB/s 2024-09-15T03:39:19.516 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:39:19.576 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-09-15T03:39:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:39:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-15 03:39:12.407550191 +0000 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-15 03:36:41.958590359 +0000 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-15 03:36:41.958590359 +0000 2024-09-15T03:39:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-15 03:36:41.958590359 +0000 2024-09-15T03:39:19.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:39:19.700 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-15T03:39:19.700 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-15T03:39:19.700 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000238845 s, 2.1 MB/s 2024-09-15T03:39:19.701 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:39:19.760 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:19.760 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:19.760 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:39:19.779 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:39:19.779 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 03:39:02.907320913 +0000 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 03:36:36.698124735 +0000 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 03:36:36.698124735 +0000 2024-09-15T03:39:19.837 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-15 03:36:36.698124735 +0000 2024-09-15T03:39:19.838 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:39:19.905 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T03:39:19.905 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T03:39:19.905 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000273553 s, 1.9 MB/s 2024-09-15T03:39:19.907 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:39:19.967 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-09-15T03:39:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:39:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:20.026 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T03:39:20.026 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 03:39:02.907320913 +0000 2024-09-15T03:39:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 03:36:37.042119688 +0000 2024-09-15T03:39:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 03:36:37.042119688 +0000 2024-09-15T03:39:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-15 03:36:37.042119688 +0000 2024-09-15T03:39:20.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:39:20.095 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T03:39:20.095 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T03:39:20.096 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000226228 s, 2.3 MB/s 2024-09-15T03:39:20.097 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:39:20.156 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-09-15T03:39:20.215 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:39:20.215 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:20.215 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-15T03:39:20.216 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:20.216 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:20.216 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 03:39:02.908320897 +0000 2024-09-15T03:39:20.216 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 03:36:37.372114846 +0000 2024-09-15T03:39:20.216 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 03:36:37.372114846 +0000 2024-09-15T03:39:20.216 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-15 03:36:37.372114846 +0000 2024-09-15T03:39:20.216 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:39:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T03:39:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T03:39:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000209896 s, 2.4 MB/s 2024-09-15T03:39:20.284 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:39:20.343 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-09-15T03:39:20.401 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:39:20.401 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:20.401 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 727 Links: 1 2024-09-15T03:39:20.401 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:20.401 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:20.401 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 03:39:02.908320897 +0000 2024-09-15T03:39:20.402 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 03:36:37.710109886 +0000 2024-09-15T03:39:20.402 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 03:36:37.710109886 +0000 2024-09-15T03:39:20.402 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-15 03:36:37.710109886 +0000 2024-09-15T03:39:20.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:39:20.467 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T03:39:20.468 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T03:39:20.468 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000218433 s, 2.3 MB/s 2024-09-15T03:39:20.469 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:39:20.527 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:20.527 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:20.527 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:39:20.547 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:39:20.547 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-09-15 03:39:07.715604311 +0000 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-09-15 03:36:36.702627784 +0000 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-09-15 03:36:36.702627784 +0000 2024-09-15T03:39:20.606 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-09-15 03:36:36.702627784 +0000 2024-09-15T03:39:20.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:39:20.673 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-09-15T03:39:20.673 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-09-15T03:39:20.673 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.00019773 s, 2.6 MB/s 2024-09-15T03:39:20.674 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:39:20.733 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-09-15T03:39:20.791 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:39:20.791 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:20.791 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T03:39:20.792 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:20.792 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:20.792 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-09-15 03:39:07.715604311 +0000 2024-09-15T03:39:20.792 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-09-15 03:36:37.040621617 +0000 2024-09-15T03:39:20.792 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-09-15 03:36:37.040621617 +0000 2024-09-15T03:39:20.792 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-09-15 03:36:37.040621617 +0000 2024-09-15T03:39:20.792 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:39:20.857 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-09-15T03:39:20.857 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-09-15T03:39:20.857 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000197368 s, 2.6 MB/s 2024-09-15T03:39:20.859 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:39:20.916 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-09-15T03:39:20.972 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:39:20.972 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:20.972 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-15T03:39:20.972 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:20.973 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:20.973 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-09-15 03:39:07.715604311 +0000 2024-09-15T03:39:20.973 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-09-15 03:36:37.388615266 +0000 2024-09-15T03:39:20.973 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-09-15 03:36:37.388615266 +0000 2024-09-15T03:39:20.973 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-09-15 03:36:37.388615266 +0000 2024-09-15T03:39:20.973 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:39:21.037 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-09-15T03:39:21.038 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-09-15T03:39:21.038 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.00020239 s, 2.5 MB/s 2024-09-15T03:39:21.039 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:39:21.097 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-09-15T03:39:21.153 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:39:21.153 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:39:21.153 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-15T03:39:21.154 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:39:21.154 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:39:21.154 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-09-15 03:39:07.716604292 +0000 2024-09-15T03:39:21.154 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-09-15 03:36:37.721609189 +0000 2024-09-15T03:39:21.154 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-09-15 03:36:37.721609189 +0000 2024-09-15T03:39:21.155 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-09-15 03:36:37.721609189 +0000 2024-09-15T03:39:21.156 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:39:21.216 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-09-15T03:39:21.216 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-09-15T03:39:21.216 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000259696 s, 2.0 MB/s 2024-09-15T03:39:21.218 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:39:21.274 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:21.274 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-15T03:39:21.274 INFO:tasks.ceph:Generating config... 2024-09-15T03:39:21.275 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T03:39:21.276 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-15T03:39:21.277 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-15T03:39:21.278 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T03:39:21.278 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T03:39:21.278 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T03:39:21.278 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T03:39:21.278 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T03:39:21.278 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-15T03:39:21.280 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-15T03:39:21.280 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-15T03:39:21.280 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T03:39:21.280 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-15T03:39:21.280 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-15T03:39:21.280 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-15T03:39:21.280 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-15T03:39:21.281 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-15T03:39:21.281 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T03:39:21.281 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-09-15T03:39:21.281 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-15T03:39:21.281 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T03:39:21.281 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T03:39:21.281 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T03:39:21.281 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T03:39:21.281 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T03:39:21.281 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T03:39:21.281 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T03:39:21.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T03:39:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T03:39:21.363 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T03:39:21.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:39:21.475 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.5'), ('mon.b', '172.21.15.64'), ('mon.c', '172.21.15.78')] 2024-09-15T03:39:21.477 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5,172.21.15.64,172.21.15.78', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-15T03:39:21.479 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:21.479 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:39:21.532 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --add b 172.21.15.64 --add c 172.21.15.78 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = octopus 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:fsid da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-09-15T03:39:21.616236+0000 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-15T03:39:21.616236+0000 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 15 (octopus) 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 3 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.b 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:2: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.c 2024-09-15T03:39:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T03:39:21.618 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:39:21.675 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID da8e668f-207e-4717-9e66-7070b1986601... 2024-09-15T03:39:21.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:39:21.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:39:21.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:39:21.723 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:39:21.760 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-09-15T03:39:21.762 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-09-15T03:39:21.763 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.5,172.21.15.64,172.21.15.78" 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: mon election default strategy = 3 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject delay max = 1 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject delay probability = 0.005 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject delay type = osd 2024-09-15T03:39:21.764 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject internal delays = 0.002 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 2500 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 120 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.765 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.766 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: debug monc = 20 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = stupid 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-09-15T03:39:21.767 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore zero block detection = True 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug reject backfill probability = 0.1 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify cached snaps = True 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify missing on start = True 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:39:21.768 INFO:teuthology.orchestra.run.smithi005.stdout: osd max backfills = 3 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd max markdown count = 1000 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock override recovery settings = True 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue = debug_random 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue cut off = debug_random 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub min interval = 60 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: osd shutdown pgref assert = True 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-15T03:39:21.769 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-09-15T03:39:21.770 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub interval = 300 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:39:21.772 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-09-15T03:39:21.773 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2024-09-15T03:39:21.773 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-09-15T03:39:21.774 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.775 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:39:21.776 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "172.21.15.5,172.21.15.64,172.21.15.78" 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: mon election default strategy = 3 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject delay max = 1 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject delay probability = 0.005 2024-09-15T03:39:21.777 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject delay type = osd 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject internal delays = 0.002 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 2500 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 120 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:39:21.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.779 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: debug monc = 20 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: bdev async discard = True 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: bdev enable discard = True 2024-09-15T03:39:21.780 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore allocator = stupid 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore zero block detection = True 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 20 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 20 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 10 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug reject backfill probability = 0.1 2024-09-15T03:39:21.781 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify cached snaps = True 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify missing on start = True 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd max backfills = 3 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd max markdown count = 1000 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock override recovery settings = True 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd op queue = debug_random 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd op queue cut off = debug_random 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub min interval = 60 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: osd shutdown pgref assert = True 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.782 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-09-15T03:39:21.783 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.784 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: mon scrub interval = 300 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-09-15T03:39:21.785 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.786 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.787 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-09-15T03:39:21.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-09-15T03:39:21.789 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.5,172.21.15.64,172.21.15.78" 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 3 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject delay max = 1 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject delay probability = 0.005 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject delay type = osd 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject internal delays = 0.002 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 2500 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-09-15T03:39:21.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 120 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-09-15T03:39:21.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-09-15T03:39:21.792 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: bdev async discard = True 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: bdev enable discard = True 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore allocator = stupid 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:39:21.793 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug reject backfill probability = 0.1 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd max backfills = 3 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub min interval = 60 2024-09-15T03:39:21.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-09-15T03:39:21.795 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:39:21.796 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-09-15T03:39:21.797 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2024-09-15T03:39:21.798 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2024-09-15T03:39:21.799 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2024-09-15T03:39:21.800 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2024-09-15T03:39:21.801 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = "172.21.15.5,172.21.15.64,172.21.15.78" 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: mon election default strategy = 3 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject delay max = 1 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject delay probability = 0.005 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject delay type = osd 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject internal delays = 0.002 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 2500 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: osd_async_recovery_min_cost = 1 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2024-09-15T03:39:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 120 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2024-09-15T03:39:21.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2024-09-15T03:39:21.804 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: debug monc = 20 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bdev async discard = True 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bdev enable discard = True 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore allocator = stupid 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore block size = 96636764160 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore fsck on mount = True 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore zero block detection = True 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluefs = 20 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluestore = 20 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-09-15T03:39:21.805 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: debug rocksdb = 10 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd full ratio = 0.9 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug reject backfill probability = 0.1 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify cached snaps = True 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify missing on start = True 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd max backfills = 3 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd max markdown count = 1000 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock override recovery settings = True 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = bluestore 2024-09-15T03:39:21.806 INFO:teuthology.orchestra.run.smithi105.stdout: osd op queue = debug_random 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: osd op queue cut off = debug_random 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub min interval = 60 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: osd shutdown pgref assert = True 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2024-09-15T03:39:21.807 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.808 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: # 1m isn't quite enough 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: mon scrub interval = 300 2024-09-15T03:39:21.809 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.b] 2024-09-15T03:39:21.810 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.c] 2024-09-15T03:39:21.813 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T03:39:21.813 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T03:39:21.902 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T03:39:21.902 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:21.902 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T03:39:21.918 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:21.918 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T03:39:21.974 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:39:21.974 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:21.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:39:21.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:39:22.050 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:39:22.050 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:22.106 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:39:22.106 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:22.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:39:22.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:39:22.149 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:39:22.150 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:22.208 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:39:22.208 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:22.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:39:22.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:39:22.249 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:39:22.249 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:22.306 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:39:22.307 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:22.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:39:22.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:39:22.345 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:39:22.346 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:22.402 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T03:39:22.402 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T03:39:22.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-15T03:39:22.453 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-15T03:39:22.456 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:39:22.518 INFO:teuthology.orchestra.run.smithi105.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:39:22.521 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T03:39:22.521 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T03:39:22.521 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T03:39:22.559 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T03:39:22.572 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T03:39:22.572 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-15T03:39:22.573 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T03:39:22.600 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:22.600 INFO:tasks.ceph:role: osd.0 2024-09-15T03:39:22.600 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:39:22.601 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:39:22.666 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:22.666 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:22.666 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:22.666 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:22.666 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:22.667 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:22.667 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:22.667 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:22.667 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:22.667 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:22.698 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-15T03:39:22.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-15T03:39:22.700 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T03:39:22.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T03:39:22.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T03:39:22.908 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:22.908 INFO:tasks.ceph:role: osd.4 2024-09-15T03:39:22.908 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:39:22.908 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:39:22.974 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:22.974 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:22.974 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:22.974 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:22.974 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:22.974 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:22.975 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:22.975 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:22.975 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:22.975 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:23.006 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-15T03:39:23.008 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-15T03:39:23.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T03:39:23.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T03:39:23.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-15T03:39:23.214 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:23.214 INFO:tasks.ceph:role: osd.8 2024-09-15T03:39:23.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:39:23.214 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:39:23.278 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:23.278 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:23.278 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:23.279 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:23.311 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-15T03:39:23.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-15T03:39:23.313 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-15T03:39:23.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-15T03:39:23.455 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-15T03:39:23.522 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:23.522 INFO:tasks.ceph:role: osd.12 2024-09-15T03:39:23.522 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:39:23.522 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-15T03:39:23.622 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-15T03:39:23.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-15T03:39:23.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-15T03:39:23.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:23.863+0000 7ffa59608440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T03:39:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:23.863+0000 7ffa59608440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T03:39:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:23.863+0000 7ffa59608440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:39:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:23.863+0000 7ffa59608440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:39:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:23.863+0000 7ffa59608440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:39:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:23.863+0000 7ffa59608440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T03:39:24.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:24.385+0000 7ffa59608440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:25.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:25.148+0000 7ffa59608440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:25.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:25.911+0000 7ffa59608440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:26.709 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T03:39:26.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:26.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:26.826+0000 7f076400c440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T03:39:26.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:26.826+0000 7f076400c440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T03:39:26.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:26.826+0000 7f076400c440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:39:26.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:26.826+0000 7f076400c440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:39:26.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:26.826+0000 7f076400c440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:39:26.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:26.826+0000 7f076400c440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T03:39:27.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:27.351+0000 7f076400c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:28.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:28.112+0000 7f076400c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:28.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:28.875+0000 7f076400c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:29.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T03:39:29.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:29.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:29.789+0000 7f6e8c721440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-15T03:39:29.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:29.790+0000 7f6e8c721440 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-15T03:39:29.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:29.790+0000 7f6e8c721440 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-09-15T03:39:29.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:29.790+0000 7f6e8c721440 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-09-15T03:39:29.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:29.790+0000 7f6e8c721440 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-09-15T03:39:29.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:29.790+0000 7f6e8c721440 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-15T03:39:30.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:30.314+0000 7f6e8c721440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:31.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:31.076+0000 7f6e8c721440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:31.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:31.839+0000 7f6e8c721440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:32.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-15T03:39:32.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:32.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:32.750+0000 7fa4ddd23440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-15T03:39:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:32.750+0000 7fa4ddd23440 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-15T03:39:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:32.750+0000 7fa4ddd23440 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-09-15T03:39:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:32.750+0000 7fa4ddd23440 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-09-15T03:39:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:32.750+0000 7fa4ddd23440 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-09-15T03:39:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:32.751+0000 7fa4ddd23440 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-15T03:39:33.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:33.277+0000 7fa4ddd23440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:34.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:34.040+0000 7fa4ddd23440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:34.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:39:34.803+0000 7fa4ddd23440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:35.604 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-15T03:39:35.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T03:39:35.962 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:35.962 INFO:tasks.ceph:role: osd.1 2024-09-15T03:39:35.962 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:39:35.963 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-15T03:39:36.565 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-15T03:39:36.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-15T03:39:36.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T03:39:36.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T03:39:36.822 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:36.822 INFO:tasks.ceph:role: osd.5 2024-09-15T03:39:36.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:39:36.822 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:36.897 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:37.345 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-15T03:39:37.347 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-15T03:39:37.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-15T03:39:37.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T03:39:37.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-15T03:39:37.564 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:37.565 INFO:tasks.ceph:role: osd.9 2024-09-15T03:39:37.565 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:39:37.565 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:37.638 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:37.639 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:37.639 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:38.133 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-15T03:39:38.135 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-15T03:39:38.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-15T03:39:38.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-15T03:39:38.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-15T03:39:38.350 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:38.350 INFO:tasks.ceph:role: osd.13 2024-09-15T03:39:38.350 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:39:38.351 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:39:38.423 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:38.424 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:38.925 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-15T03:39:38.927 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-15T03:39:38.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-15T03:39:39.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-15T03:39:39.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:39.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.185+0000 7f103892a440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T03:39:39.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.185+0000 7f103892a440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T03:39:39.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.185+0000 7f103892a440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:39:39.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.185+0000 7f103892a440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:39:39.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.185+0000 7f103892a440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:39:39.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.187+0000 7f103892a440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T03:39:39.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:39.712+0000 7f103892a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:40.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:40.481+0000 7f103892a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:41.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:41.243+0000 7f103892a440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:42.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T03:39:42.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:42.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.159+0000 7f097391d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T03:39:42.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.159+0000 7f097391d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T03:39:42.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.159+0000 7f097391d440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:39:42.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.159+0000 7f097391d440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:39:42.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.159+0000 7f097391d440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:39:42.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.159+0000 7f097391d440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T03:39:42.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:42.684+0000 7f097391d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:43.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:43.452+0000 7f097391d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:44.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:44.215+0000 7f097391d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:45.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T03:39:45.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:45.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.140+0000 7faded926440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-15T03:39:45.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.141+0000 7faded926440 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-15T03:39:45.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.141+0000 7faded926440 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-09-15T03:39:45.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.141+0000 7faded926440 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-09-15T03:39:45.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.141+0000 7faded926440 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-09-15T03:39:45.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.141+0000 7faded926440 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-15T03:39:45.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:45.668+0000 7faded926440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:46.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:46.432+0000 7faded926440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:47.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:47.199+0000 7faded926440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:47.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-15T03:39:48.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:48.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.115+0000 7f7ab2a75440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-15T03:39:48.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.115+0000 7f7ab2a75440 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-15T03:39:48.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.115+0000 7f7ab2a75440 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-09-15T03:39:48.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.115+0000 7f7ab2a75440 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-09-15T03:39:48.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.115+0000 7f7ab2a75440 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-09-15T03:39:48.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.116+0000 7f7ab2a75440 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-15T03:39:48.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:48.644+0000 7f7ab2a75440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:49.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:49.406+0000 7f7ab2a75440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-15T03:39:50.173+0000 7f7ab2a75440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:50.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-15T03:39:51.000 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T03:39:51.030 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:51.030 INFO:tasks.ceph:role: osd.2 2024-09-15T03:39:51.030 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:39:51.030 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:51.107 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:51.108 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:51.108 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:51.608 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T03:39:51.610 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T03:39:51.610 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-15T03:39:51.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T03:39:51.780 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-15T03:39:51.842 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:51.842 INFO:tasks.ceph:role: osd.6 2024-09-15T03:39:51.843 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:39:51.843 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:51.915 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:52.369 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T03:39:52.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T03:39:52.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-15T03:39:52.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-15T03:39:52.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-15T03:39:52.593 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:52.594 INFO:tasks.ceph:role: osd.10 2024-09-15T03:39:52.594 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:39:52.594 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:39:52.665 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:52.666 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:53.168 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T03:39:53.170 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T03:39:53.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-15T03:39:53.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-15T03:39:53.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-15T03:39:53.392 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-15T03:39:53.392 INFO:tasks.ceph:role: osd.14 2024-09-15T03:39:53.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:39:53.392 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:39:53.465 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:39:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:39:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:39:53.466 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:39:53.973 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T03:39:53.976 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T03:39:53.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-15T03:39:54.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-15T03:39:54.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:54.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.249+0000 7f350c557440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T03:39:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.249+0000 7f350c557440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T03:39:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.249+0000 7f350c557440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:39:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.249+0000 7f350c557440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:39:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.249+0000 7f350c557440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:39:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.250+0000 7f350c557440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T03:39:54.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:54.777+0000 7f350c557440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:55.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:55.541+0000 7f350c557440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:56.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:56.313+0000 7f350c557440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:57.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T03:39:57.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:39:57.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.229+0000 7f04fea31440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-15T03:39:57.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.229+0000 7f04fea31440 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-15T03:39:57.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.229+0000 7f04fea31440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T03:39:57.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.229+0000 7f04fea31440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T03:39:57.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.229+0000 7f04fea31440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-15T03:39:57.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.230+0000 7f04fea31440 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-15T03:39:57.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:57.754+0000 7f04fea31440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:58.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:58.521+0000 7f04fea31440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:39:59.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:39:59.287+0000 7f04fea31440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:00.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-15T03:40:00.112 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:00.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.201+0000 7f85a7123440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-15T03:40:00.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.201+0000 7f85a7123440 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-15T03:40:00.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.201+0000 7f85a7123440 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-09-15T03:40:00.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.201+0000 7f85a7123440 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-09-15T03:40:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.201+0000 7f85a7123440 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-09-15T03:40:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.202+0000 7f85a7123440 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-15T03:40:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:00.729+0000 7f85a7123440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:01.493+0000 7f85a7123440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:02.255+0000 7f85a7123440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:03.059 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-15T03:40:03.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.165+0000 7f03ba408440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-15T03:40:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.165+0000 7f03ba408440 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-15T03:40:03.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.165+0000 7f03ba408440 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-09-15T03:40:03.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.165+0000 7f03ba408440 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-09-15T03:40:03.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.165+0000 7f03ba408440 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-09-15T03:40:03.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.166+0000 7f03ba408440 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-15T03:40:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:03.688+0000 7f03ba408440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:04.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:04.458+0000 7f03ba408440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T03:40:05.221+0000 7f03ba408440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:06.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-15T03:40:06.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T03:40:06.123 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T03:40:06.123 INFO:tasks.ceph:role: osd.3 2024-09-15T03:40:06.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:40:06.124 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:40:06.200 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:40:06.200 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:40:06.200 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:40:06.200 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:40:06.200 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:40:06.200 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:40:06.201 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:40:06.201 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:40:06.201 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:40:06.201 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:40:06.721 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-09-15T03:40:06.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-09-15T03:40:06.725 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T03:40:06.853 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T03:40:06.888 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-15T03:40:06.958 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T03:40:06.958 INFO:tasks.ceph:role: osd.7 2024-09-15T03:40:06.958 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:40:06.959 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:40:07.033 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:40:07.034 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:40:07.489 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-09-15T03:40:07.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-09-15T03:40:07.492 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-15T03:40:07.587 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-15T03:40:07.618 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-15T03:40:07.685 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T03:40:07.686 INFO:tasks.ceph:role: osd.11 2024-09-15T03:40:07.686 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:40:07.686 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:40:07.759 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:40:07.759 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:40:07.759 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:40:07.759 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:40:07.760 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:40:07.760 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:40:07.760 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:40:07.760 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:40:07.760 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:40:07.760 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:40:08.255 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-09-15T03:40:08.257 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-09-15T03:40:08.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-15T03:40:08.412 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-15T03:40:08.444 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-15T03:40:08.510 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-15T03:40:08.510 INFO:tasks.ceph:role: osd.15 2024-09-15T03:40:08.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:40:08.511 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:40:08.584 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:40:08.585 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:40:09.103 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-09-15T03:40:09.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-09-15T03:40:09.106 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-15T03:40:09.242 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-15T03:40:09.276 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:09.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.386+0000 7fe9a5b27440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T03:40:09.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.386+0000 7fe9a5b27440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T03:40:09.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.386+0000 7fe9a5b27440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:40:09.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.386+0000 7fe9a5b27440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:40:09.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.386+0000 7fe9a5b27440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:40:09.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.386+0000 7fe9a5b27440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T03:40:09.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:09.911+0000 7fe9a5b27440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:10.678 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:10.677+0000 7fe9a5b27440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:11.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:11.439+0000 7fe9a5b27440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:12.240 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T03:40:12.269 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:12.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.354+0000 7f252672f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-15T03:40:12.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.355+0000 7f252672f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-15T03:40:12.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.355+0000 7f252672f440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T03:40:12.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.355+0000 7f252672f440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T03:40:12.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.355+0000 7f252672f440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-15T03:40:12.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.355+0000 7f252672f440 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-15T03:40:12.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:12.878+0000 7f252672f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:13.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:13.643+0000 7f252672f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:14.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:14.406+0000 7f252672f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:15.200 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-15T03:40:15.228 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:15.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.312+0000 7f9e13a0f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-15T03:40:15.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.313+0000 7f9e13a0f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-15T03:40:15.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.313+0000 7f9e13a0f440 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-09-15T03:40:15.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.313+0000 7f9e13a0f440 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-09-15T03:40:15.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.313+0000 7f9e13a0f440 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-09-15T03:40:15.315 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.313+0000 7f9e13a0f440 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-15T03:40:15.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:15.840+0000 7f9e13a0f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:16.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:16.604+0000 7f9e13a0f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:17.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:17.367+0000 7f9e13a0f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:18.164 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-15T03:40:18.189 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:18.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.269+0000 7f0b02323440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-15T03:40:18.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.269+0000 7f0b02323440 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-15T03:40:18.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.269+0000 7f0b02323440 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-09-15T03:40:18.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.269+0000 7f0b02323440 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-09-15T03:40:18.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.269+0000 7f0b02323440 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-09-15T03:40:18.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.270+0000 7f0b02323440 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-15T03:40:18.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:18.793+0000 7f0b02323440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:19.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:19.562+0000 7f0b02323440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:20.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:20.331+0000 7f0b02323440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:40:21.129 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-15T03:40:21.199 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T03:40:21.199 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:21.200 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-15T03:40:21.233 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:21.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T03:40:21.303 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:21.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T03:40:21.372 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:21.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-15T03:40:21.440 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:21.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-15T03:40:21.508 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:21.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T03:40:21.541 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:21.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T03:40:21.612 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:21.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-15T03:40:21.682 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:21.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-15T03:40:21.751 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:21.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T03:40:21.781 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:21.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-15T03:40:21.852 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:21.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-15T03:40:21.921 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:21.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-15T03:40:21.992 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:21.992 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T03:40:22.026 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:22.027 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T03:40:22.107 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:22.107 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-15T03:40:22.177 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:22.177 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-15T03:40:22.244 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:22.245 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-15T03:40:22.312 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:22.312 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T03:40:22.369 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T03:40:22.370 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:40:22.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:40:22.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.y] 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDqVuZmj7YFGxAAE2wju5pKOP5r5I2XQEINIg== 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDrVuZmgQKMMxAAQGYNot6KSc4lygz9pbY9DA== 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDuVuZmjJlVMRAAhlt2iJgcFDYliT9R/qSJTA== 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.8] 2024-09-15T03:40:22.399 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDxVuZmqawrLxAAX98hMGzdGSBKA7mrKdNSWg== 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.12] 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQD0VuZmiDzULBAA/dwEFrK1B1WMTXVPKKvIsg== 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQD7VuZmRroZCxAAA15BNWBBUVR3qhd5yow3iA== 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQD+VuZmB1OYCRAA67NYHCTuTJvK5OTQg0DWGQ== 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.9] 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQABV+ZmMcKCCBAAiMgWT2hoBNuv5iwv5OaUZA== 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.13] 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAEV+Zmf83tBhAAJpYy6IPUmNhUj/K/2fCWXw== 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-09-15T03:40:22.400 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAKV+ZmeDvsDhAA9DzcaMIpqVty7mdm6V43yQ== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.6] 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQANV+Zmwwi2DRAAQMNnuSidFFWf1+DpVaOf1Q== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.10] 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAQV+ZmBTgRDBAAzulqEExDjRrLv3tqmxiZWg== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.14] 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQATV+ZmymnlCRAAcZkqVPungxfLzcCVy/6S2g== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDqVuZmJpDiHhAAVYUcCU2Ca+GAdoSUyC1FqQ== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAZV+Zm1mcRFxAAmZTrpxAxhJCqO0s1+0CSHA== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.7] 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAcV+ZmNKs0FRAAk7APWS4d2BPiVTlzP0tDew== 2024-09-15T03:40:22.401 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.11] 2024-09-15T03:40:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAfV+ZmZYayEhAAM1GSF2w6QhL94v+xm0XX4Q== 2024-09-15T03:40:22.402 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.15] 2024-09-15T03:40:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAiV+ZmcRoeEBAASvFqSIsD23Q9zoCbdAgehQ== 2024-09-15T03:40:22.402 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-09-15T03:40:22.402 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDqVuZm3jhTIRAAP4pWrr1y3lRIJhIyo14BXA== 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.y] 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDqVuZmj7YFGxAAE2wju5pKOP5r5I2XQEINIg== 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDrVuZmgQKMMxAAQGYNot6KSc4lygz9pbY9DA== 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDuVuZmjJlVMRAAhlt2iJgcFDYliT9R/qSJTA== 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.8] 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDxVuZmqawrLxAAX98hMGzdGSBKA7mrKdNSWg== 2024-09-15T03:40:22.403 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.12] 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQD0VuZmiDzULBAA/dwEFrK1B1WMTXVPKKvIsg== 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQD7VuZmRroZCxAAA15BNWBBUVR3qhd5yow3iA== 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQD+VuZmB1OYCRAA67NYHCTuTJvK5OTQg0DWGQ== 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.9] 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQABV+ZmMcKCCBAAiMgWT2hoBNuv5iwv5OaUZA== 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.13] 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAEV+Zmf83tBhAAJpYy6IPUmNhUj/K/2fCWXw== 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-09-15T03:40:22.404 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAKV+ZmeDvsDhAA9DzcaMIpqVty7mdm6V43yQ== 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.6] 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQANV+Zmwwi2DRAAQMNnuSidFFWf1+DpVaOf1Q== 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.10] 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAQV+ZmBTgRDBAAzulqEExDjRrLv3tqmxiZWg== 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.14] 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQATV+ZmymnlCRAAcZkqVPungxfLzcCVy/6S2g== 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDqVuZmJpDiHhAAVYUcCU2Ca+GAdoSUyC1FqQ== 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAZV+Zm1mcRFxAAmZTrpxAxhJCqO0s1+0CSHA== 2024-09-15T03:40:22.405 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.7] 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAcV+ZmNKs0FRAAk7APWS4d2BPiVTlzP0tDew== 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.11] 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAfV+ZmZYayEhAAM1GSF2w6QhL94v+xm0XX4Q== 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.15] 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAiV+ZmcRoeEBAASvFqSIsD23Q9zoCbdAgehQ== 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-09-15T03:40:22.406 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDqVuZm3jhTIRAAP4pWrr1y3lRIJhIyo14BXA== 2024-09-15T03:40:22.407 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-09-15T03:40:22.407 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDqVuZmj7YFGxAAE2wju5pKOP5r5I2XQEINIg== 2024-09-15T03:40:22.407 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-09-15T03:40:22.407 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDrVuZmgQKMMxAAQGYNot6KSc4lygz9pbY9DA== 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDuVuZmjJlVMRAAhlt2iJgcFDYliT9R/qSJTA== 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.8] 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDxVuZmqawrLxAAX98hMGzdGSBKA7mrKdNSWg== 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.12] 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQD0VuZmiDzULBAA/dwEFrK1B1WMTXVPKKvIsg== 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQD7VuZmRroZCxAAA15BNWBBUVR3qhd5yow3iA== 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQD+VuZmB1OYCRAA67NYHCTuTJvK5OTQg0DWGQ== 2024-09-15T03:40:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.9] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQABV+ZmMcKCCBAAiMgWT2hoBNuv5iwv5OaUZA== 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.13] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAEV+Zmf83tBhAAJpYy6IPUmNhUj/K/2fCWXw== 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAKV+ZmeDvsDhAA9DzcaMIpqVty7mdm6V43yQ== 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQANV+Zmwwi2DRAAQMNnuSidFFWf1+DpVaOf1Q== 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.10] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAQV+ZmBTgRDBAAzulqEExDjRrLv3tqmxiZWg== 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.14] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQATV+ZmymnlCRAAcZkqVPungxfLzcCVy/6S2g== 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-09-15T03:40:22.409 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDqVuZmJpDiHhAAVYUcCU2Ca+GAdoSUyC1FqQ== 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAZV+Zm1mcRFxAAmZTrpxAxhJCqO0s1+0CSHA== 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAcV+ZmNKs0FRAAk7APWS4d2BPiVTlzP0tDew== 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.11] 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAfV+ZmZYayEhAAM1GSF2w6QhL94v+xm0XX4Q== 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.15] 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAiV+ZmcRoeEBAASvFqSIsD23Q9zoCbdAgehQ== 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-09-15T03:40:22.410 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDqVuZm3jhTIRAAP4pWrr1y3lRIJhIyo14BXA== 2024-09-15T03:40:22.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:40:22.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:40:22.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:40:22.505 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.563 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.717 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.818 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.915 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.965 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:22.970 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:40:23.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:40:23.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:40:23.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.172 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.273 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:40:23.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:40:23.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:40:23.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:40:23.422 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T03:40:23.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T03:40:23.448 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:40:23.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T03:40:23.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T03:40:23.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:40:23.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T03:40:23.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T03:40:23.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:40:23.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T03:40:23.850 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:23.852 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:23.855 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:40:23.905 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T03:40:23.905 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:40:23.906 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:40:23.909 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:40:23.909 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:40:23.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:40:23.911 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:40:23.911 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:40:23.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:40:23.947 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:40:23.947 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T03:40:23.948 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-15T03:40:23.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-15T03:40:23.954 INFO:tasks.ceph.mgr.y:Started 2024-09-15T03:40:23.954 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T03:40:23.954 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T03:40:23.957 INFO:tasks.ceph.mgr.x:Started 2024-09-15T03:40:23.957 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T03:40:23.957 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T03:40:30.338 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-09-15T03:40:30.349 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T03:40:30.350 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:30.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T03:40:30.366 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:30.365+0000 7fd981dc3ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:40:30.372 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:30.372+0000 7f08c960fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:40:30.376 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:30.376 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T03:40:30.406 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:30.405+0000 7fd981dc3ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:40:30.414 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:30.413+0000 7f08c960fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:40:30.439 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:30.439 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-15T03:40:30.454 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:30.453+0000 7fd981dc3ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:40:30.466 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:30.466+0000 7f08c960fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:40:30.491 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-15T03:40:30.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-15T03:40:30.557 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:30.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T03:40:30.588 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:30.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T03:40:30.593 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:30.592+0000 7fd981dc3ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:40:30.644 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:30.643+0000 7f08c960fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:40:30.657 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:30.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-15T03:40:30.698 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:30.697+0000 7fd981dc3ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:40:30.726 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-15T03:40:30.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-15T03:40:30.755 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:30.755+0000 7f08c960fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:40:30.794 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:30.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T03:40:30.824 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:30.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-15T03:40:30.892 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:30.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-15T03:40:30.960 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T03:40:30.960 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-15T03:40:30.987 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:30.986+0000 7fd981dc3ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:40:31.026 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:31.026 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T03:40:31.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:31.034+0000 7fd981dc3ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:40:31.051 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:31.051 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-15T03:40:31.055 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:31.055+0000 7f08c960fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:40:31.110 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:31.110+0000 7f08c960fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:40:31.115 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:31.115 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-15T03:40:31.171 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:31.170+0000 7fd981dc3ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:40:31.179 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-15T03:40:31.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-15T03:40:31.244 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 2efef372-d786-40e5-96a1-10f399314942 0 2024-09-15T03:40:31.247 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:31.247+0000 7f08c960fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:40:31.477 INFO:teuthology.orchestra.run.smithi105.stdout:0 2024-09-15T03:40:31.488 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new d77b5a7a-5228-4bed-b89b-799b0d0b9491 1 2024-09-15T03:40:31.662 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-09-15T03:40:31.673 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new e81da5de-c743-4cf0-a36a-beee634f01da 2 2024-09-15T03:40:31.847 INFO:teuthology.orchestra.run.smithi105.stdout:2 2024-09-15T03:40:31.857 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new dae3f484-9f24-4f24-96c3-253465d0d369 3 2024-09-15T03:40:32.031 INFO:teuthology.orchestra.run.smithi105.stdout:3 2024-09-15T03:40:32.042 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 7a2d4b3d-e398-4010-8952-4e5207e005b2 4 2024-09-15T03:40:32.163 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.156+0000 7fd981dc3ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:40:32.202 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.201+0000 7f08c960fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:40:32.206 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.206+0000 7fd981dc3ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:40:32.214 INFO:teuthology.orchestra.run.smithi105.stdout:4 2024-09-15T03:40:32.226 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new b757a5a4-3278-42ec-b43f-6c08bfc15f06 5 2024-09-15T03:40:32.254 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.254+0000 7f08c960fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:40:32.398 INFO:teuthology.orchestra.run.smithi105.stdout:5 2024-09-15T03:40:32.408 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 96572944-7f1f-4461-a818-89bd0c4e2faf 6 2024-09-15T03:40:32.596 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.595+0000 7fd981dc3ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T03:40:32.598 INFO:teuthology.orchestra.run.smithi105.stdout:6 2024-09-15T03:40:32.609 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new e698cf1c-5b5d-43a2-b44a-3d30af6f5917 7 2024-09-15T03:40:32.634 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.634+0000 7f08c960fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T03:40:32.683 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.681+0000 7fd981dc3ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:40:32.719 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.718+0000 7f08c960fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:40:32.773 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.770+0000 7fd981dc3ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:40:32.810 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.810+0000 7f08c960fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:40:32.815 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.814+0000 7fd981dc3ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:40:32.857 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.856+0000 7f08c960fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:40:32.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:40:32.903+0000 7f9ca37fe640 0 --2- 172.21.15.105:0/504873466 >> v2:172.21.15.5:3300/0 conn(0x7f9ca4111e40 0x7f9ca4112210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:40:32.907 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.906+0000 7fd981dc3ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T03:40:32.955 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:32.954+0000 7f08c960fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T03:40:32.961 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:32.960+0000 7fd981dc3ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:40:32.986 INFO:teuthology.orchestra.run.smithi105.stdout:7 2024-09-15T03:40:32.997 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 12a65fa6-1fbd-4900-88c9-69695141bc1b 8 2024-09-15T03:40:33.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.005+0000 7fd981dc3ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:40:33.009 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.009+0000 7f08c960fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:40:33.054 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.054+0000 7f08c960fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:40:33.103 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.102+0000 7fd981dc3ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:40:33.151 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.151+0000 7f08c960fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:40:33.153 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.152+0000 7fd981dc3ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:40:33.172 INFO:teuthology.orchestra.run.smithi105.stdout:8 2024-09-15T03:40:33.182 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4 9 2024-09-15T03:40:33.202 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.202+0000 7f08c960fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:40:33.204 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.203+0000 7fd981dc3ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:40:33.254 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.254+0000 7f08c960fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:40:33.274 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.273+0000 7fd981dc3ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:40:33.319 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.318+0000 7fd981dc3ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:40:33.323 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.323+0000 7f08c960fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:40:33.363 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:33.362+0000 7fd981dc3ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:40:33.368 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.367+0000 7f08c960fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:40:33.380 INFO:teuthology.orchestra.run.smithi105.stdout:9 2024-09-15T03:40:33.391 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 91199e3c-694d-4888-859b-b76feac7c96d 10 2024-09-15T03:40:33.412 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:33.411+0000 7f08c960fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:40:33.565 INFO:teuthology.orchestra.run.smithi105.stdout:10 2024-09-15T03:40:33.575 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 4d6ca3aa-f880-4673-8d66-21d438dd4b61 11 2024-09-15T03:40:33.748 INFO:teuthology.orchestra.run.smithi105.stdout:11 2024-09-15T03:40:33.760 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 5d4556fd-7659-4e2b-b60b-e44eca29c3e6 12 2024-09-15T03:40:33.934 INFO:teuthology.orchestra.run.smithi105.stdout:12 2024-09-15T03:40:33.944 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 78791966-b232-400f-9c30-cabebfa73dc0 13 2024-09-15T03:40:34.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:34.003+0000 7fd981dc3ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:40:34.054 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-09-15T03:40:34.054+0000 7f08c960fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:40:34.120 INFO:teuthology.orchestra.run.smithi105.stdout:13 2024-09-15T03:40:34.131 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 4a275057-7e86-452f-958d-80263088db83 14 2024-09-15T03:40:34.301 INFO:teuthology.orchestra.run.smithi105.stdout:14 2024-09-15T03:40:34.311 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 18589d26-e121-4e72-bf6e-24a4a37c49c5 15 2024-09-15T03:40:34.481 INFO:teuthology.orchestra.run.smithi105.stdout:15 2024-09-15T03:40:34.491 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:40:34.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:40:34.493 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:40:34.494 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T03:40:34.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T03:40:34.496 INFO:tasks.ceph.osd.4:Started 2024-09-15T03:40:34.496 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-15T03:40:34.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-15T03:40:34.498 INFO:tasks.ceph.osd.8:Started 2024-09-15T03:40:34.498 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-15T03:40:34.498 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-15T03:40:34.501 INFO:tasks.ceph.osd.12:Started 2024-09-15T03:40:34.501 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:40:34.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:40:34.504 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:40:34.504 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T03:40:34.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T03:40:34.506 INFO:tasks.ceph.osd.5:Started 2024-09-15T03:40:34.506 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-15T03:40:34.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-15T03:40:34.508 INFO:tasks.ceph.osd.9:Started 2024-09-15T03:40:34.509 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-15T03:40:34.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-15T03:40:34.511 INFO:tasks.ceph.osd.13:Started 2024-09-15T03:40:34.511 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T03:40:34.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T03:40:34.515 INFO:tasks.ceph.osd.2:Started 2024-09-15T03:40:34.516 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-15T03:40:34.516 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-15T03:40:34.519 INFO:tasks.ceph.osd.6:Started 2024-09-15T03:40:34.519 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-15T03:40:34.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-15T03:40:34.522 INFO:tasks.ceph.osd.10:Started 2024-09-15T03:40:34.523 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-15T03:40:34.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-15T03:40:34.525 INFO:tasks.ceph.osd.14:Started 2024-09-15T03:40:34.525 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:40:34.525 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:40:34.528 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:40:34.528 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-15T03:40:34.528 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-15T03:40:34.530 INFO:tasks.ceph.osd.7:Started 2024-09-15T03:40:34.530 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-15T03:40:34.530 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-15T03:40:34.532 INFO:tasks.ceph.osd.11:Started 2024-09-15T03:40:34.532 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-15T03:40:34.532 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-15T03:40:34.535 INFO:tasks.ceph.osd.15:Started 2024-09-15T03:40:34.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:40:34.720 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:34.720 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":18,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:40:34.478574+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:40:34.478574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:40:34.731 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T03:40:34.732 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T03:40:35.022 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:35.020+0000 7fd9662c9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T03:40:35.092 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:35.091+0000 7febe6812440 -1 Falling back to public interface 2024-09-15T03:40:35.097 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:40:35.096+0000 7f343771d440 -1 Falling back to public interface 2024-09-15T03:40:35.100 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:40:35.099+0000 7fbcca418440 -1 Falling back to public interface 2024-09-15T03:40:35.109 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:40:35.108+0000 7fb44bc5a440 -1 Falling back to public interface 2024-09-15T03:40:35.111 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:40:35.110+0000 7f9cafc16440 -1 Falling back to public interface 2024-09-15T03:40:35.115 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:35.114+0000 7fcb49408440 -1 Falling back to public interface 2024-09-15T03:40:35.116 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:35.114+0000 7fab20224440 -1 Falling back to public interface 2024-09-15T03:40:35.116 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:35.114+0000 7fab1e1fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2024-09-15T03:40:35.117 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:40:35.115+0000 7f0ddd12f440 -1 Falling back to public interface 2024-09-15T03:40:35.119 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:35.117+0000 7fa7d5608440 -1 Falling back to public interface 2024-09-15T03:40:35.120 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:40:35.118+0000 7f2f59935440 -1 Falling back to public interface 2024-09-15T03:40:35.123 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:40:35.121+0000 7f7a8000d440 -1 Falling back to public interface 2024-09-15T03:40:35.124 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:35.121+0000 7fbb3ed2f440 -1 Falling back to public interface 2024-09-15T03:40:35.124 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:40:35.122+0000 7feee3721440 -1 Falling back to public interface 2024-09-15T03:40:35.125 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:35.122+0000 7f5478e15440 -1 Falling back to public interface 2024-09-15T03:40:35.125 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:35.123+0000 7f5476dfd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6811/0: (98) Address already in use 2024-09-15T03:40:35.126 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:35.124+0000 7f5476dfd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6818/0: (98) Address already in use 2024-09-15T03:40:35.128 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:35.126+0000 7fde13e1e440 -1 Falling back to public interface 2024-09-15T03:40:35.129 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:40:35.127+0000 7fc29401a440 -1 Falling back to public interface 2024-09-15T03:40:35.226 INFO:tasks.ceph.mgr.x.smithi105.stderr:2024-09-15T03:40:35.224+0000 7fd9662c9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T03:40:36.343 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:40:36.626 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-15T03:40:36.626 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":18,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:40:34.478574+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:40:34.478574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:40:36.637 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-15T03:40:38.541 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:40:38.539+0000 7f0ddd12f440 -1 osd.6 0 log_to_monitors true 2024-09-15T03:40:38.560 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:38.559+0000 7febe6812440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:40:38.587 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:40:38.585+0000 7fc29401a440 -1 osd.15 0 log_to_monitors true 2024-09-15T03:40:38.599 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:40:38.599+0000 7fbcca418440 -1 osd.4 0 log_to_monitors true 2024-09-15T03:40:38.608 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:38.606+0000 7fcb49408440 -1 osd.9 0 log_to_monitors true 2024-09-15T03:40:38.612 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:38.610+0000 7fbb3ed2f440 -1 osd.14 0 log_to_monitors true 2024-09-15T03:40:38.612 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:40:38.611+0000 7f9cafc16440 -1 osd.5 0 log_to_monitors true 2024-09-15T03:40:38.621 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:40:38.619+0000 7f7a8000d440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:40:38.630 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:38.628+0000 7fa7d5608440 -1 osd.13 0 log_to_monitors true 2024-09-15T03:40:38.632 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:40:38.632+0000 7f343771d440 -1 osd.8 0 log_to_monitors true 2024-09-15T03:40:38.633 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:38.631+0000 7fde13e1e440 -1 osd.10 0 log_to_monitors true 2024-09-15T03:40:38.637 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:40:38.635+0000 7f2f59935440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:40:38.641 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:40:38.639+0000 7feee3721440 -1 osd.11 0 log_to_monitors true 2024-09-15T03:40:38.643 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:40:38.643+0000 7fb44bc5a440 -1 osd.12 0 log_to_monitors true 2024-09-15T03:40:38.645 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:38.643+0000 7fab20224440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:40:38.658 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:38.656+0000 7f5478e15440 -1 osd.7 0 log_to_monitors true 2024-09-15T03:40:39.608 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:40:39.606+0000 7fc28e7f6640 -1 osd.15 0 waiting for initial osdmap 2024-09-15T03:40:39.611 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:40:39.610+0000 7f0dd93f9640 -1 osd.6 0 waiting for initial osdmap 2024-09-15T03:40:39.614 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:39.614+0000 7febe0ff6640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T03:40:39.615 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:40:39.613+0000 7fc2877e8640 -1 osd.15 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:39.616 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:40:39.615+0000 7f0dd0be8640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:39.623 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:39.622+0000 7febd9fe8640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.617 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:40:40.616+0000 7fbcc4bf6640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T03:40:40.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:40:40.618+0000 7f9caa3f6640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T03:40:40.619 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:40.618+0000 7fa7cfdf6640 -1 osd.13 0 waiting for initial osdmap 2024-09-15T03:40:40.619 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:40.618+0000 7fab1a9f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T03:40:40.620 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:40:40.619+0000 7f34321f6640 -1 osd.8 0 waiting for initial osdmap 2024-09-15T03:40:40.620 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:40.618+0000 7fbb3aff9640 -1 osd.14 0 waiting for initial osdmap 2024-09-15T03:40:40.620 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:40:40.618+0000 7f7a7a7f6640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T03:40:40.622 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:40.621+0000 7fcb453f9640 -1 osd.9 0 waiting for initial osdmap 2024-09-15T03:40:40.622 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:40:40.621+0000 7f2f55bf9640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T03:40:40.623 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:40.622+0000 7fde0fdf9640 -1 osd.10 0 waiting for initial osdmap 2024-09-15T03:40:40.625 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:40.624+0000 7fbb327d7640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.626 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:40:40.625+0000 7fb446df6640 -1 osd.12 0 waiting for initial osdmap 2024-09-15T03:40:40.626 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:40.624+0000 7f5474df9640 -1 osd.7 0 waiting for initial osdmap 2024-09-15T03:40:40.626 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:40:40.624+0000 7f7a737e8640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.626 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:40:40.625+0000 7feede1f6640 -1 osd.11 0 waiting for initial osdmap 2024-09-15T03:40:40.627 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:40:40.627+0000 7f342b1e8640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.628 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:40:40.627+0000 7fbcbdbe8640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.630 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:40.629+0000 7fa7c8de8640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.630 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:40.629+0000 7fab139e8640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.632 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:40.631+0000 7fcb3cbe8640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.632 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:40:40.631+0000 7f2f4d3e8640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.633 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:40.632+0000 7fde075e8640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.633 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:40:40.632+0000 7fb43fc84640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.634 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:40.632+0000 7f546c5e8640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.634 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:40:40.632+0000 7feed71d7640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:40.636 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:40:40.635+0000 7f9ca33e8640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:40:44.251 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:40:44.537 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-15T03:40:44.537 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":24,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:40:43.632009+0000","last_up_change":"2024-09-15T03:40:41.615545+0000","last_in_change":"2024-09-15T03:40:34.478574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":33113},{"type":"v1","addr":"172.21.15.105:6803","nonce":33113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":33113},{"type":"v1","addr":"172.21.15.105:6805","nonce":33113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":33113},{"type":"v1","addr":"172.21.15.105:6809","nonce":33113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":33113},{"type":"v1","addr":"172.21.15.105:6807","nonce":33113}]},"public_addr":"172.21.15.105:6803/33113","cluster_addr":"172.21.15.105:6805/33113","heartbeat_back_addr":"172.21.15.105:6809/33113","heartbeat_front_addr":"172.21.15.105:6807/33113","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.532883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.696957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.621678+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.668299+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.560569+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.593272+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.525540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.677822+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.597079+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.594754+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.639694+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.691925+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.673878+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.655432+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.605739+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.574974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:40:44.547 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-15T03:40:44.547 INFO:tasks.ceph:Creating RBD pool 2024-09-15T03:40:44.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T03:40:45.648 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2024-09-15T03:40:45.661 DEBUG:teuthology.orchestra.run.smithi005:> rbd --cluster ceph pool init rbd 2024-09-15T03:40:48.673 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T03:40:48.674 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T03:40:48.674 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T03:40:48.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T03:40:48.675 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T03:40:49.008 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:49.019 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":32210},{"type":"v1","addr":"172.21.15.105:6801","nonce":32210}]},"active_addr":"172.21.15.105:6801/32210","active_change":"2024-09-15T03:40:34.006719+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4109,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":2055232518}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":3036622776}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1675871746}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1309982864}]}]} 2024-09-15T03:40:49.024 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T03:40:49.024 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:40:49.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:40:49.304 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:49.305 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":29,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:40:48.655606+0000","last_up_change":"2024-09-15T03:40:41.615545+0000","last_in_change":"2024-09-15T03:40:34.478574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":33113},{"type":"v1","addr":"172.21.15.105:6803","nonce":33113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":33113},{"type":"v1","addr":"172.21.15.105:6805","nonce":33113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":33113},{"type":"v1","addr":"172.21.15.105:6809","nonce":33113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":33113},{"type":"v1","addr":"172.21.15.105:6807","nonce":33113}]},"public_addr":"172.21.15.105:6803/33113","cluster_addr":"172.21.15.105:6805/33113","heartbeat_back_addr":"172.21.15.105:6809/33113","heartbeat_front_addr":"172.21.15.105:6807/33113","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.532883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.696957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.621678+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.668299+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.560569+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.593272+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.525540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.677822+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.597079+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.594754+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.639694+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.691925+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.673878+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.655432+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.605739+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.574974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:40:49.317 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:40:49.317 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:40:49.596 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:49.596 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":29,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:40:48.655606+0000","last_up_change":"2024-09-15T03:40:41.615545+0000","last_in_change":"2024-09-15T03:40:34.478574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":33113},{"type":"v1","addr":"172.21.15.105:6803","nonce":33113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":33113},{"type":"v1","addr":"172.21.15.105:6805","nonce":33113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":33113},{"type":"v1","addr":"172.21.15.105:6809","nonce":33113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":33113},{"type":"v1","addr":"172.21.15.105:6807","nonce":33113}]},"public_addr":"172.21.15.105:6803/33113","cluster_addr":"172.21.15.105:6805/33113","heartbeat_back_addr":"172.21.15.105:6809/33113","heartbeat_front_addr":"172.21.15.105:6807/33113","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.532883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.696957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.621678+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.668299+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.560569+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.593272+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.525540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.677822+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.597079+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.594754+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.639694+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.691925+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.673878+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.655432+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.605739+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.574974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:40:49.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:40:49.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:40:49.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:40:49.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:40:49.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:40:49.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:40:49.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T03:40:49.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T03:40:49.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-15T03:40:49.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-15T03:40:49.613 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-15T03:40:49.613 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-15T03:40:49.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-15T03:40:49.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-15T03:40:49.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-15T03:40:49.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-15T03:40:50.025 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T03:40:50.056 INFO:teuthology.orchestra.run.smithi005.stdout:94489280517 2024-09-15T03:40:50.056 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:40:50.057 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-15T03:40:50.065 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.066 INFO:teuthology.orchestra.run.smithi005.stdout:90194313221 2024-09-15T03:40:50.066 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-15T03:40:50.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:40:50.077 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.077 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:50.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:40:50.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:40:50.088 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:50.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:40:50.095 INFO:teuthology.orchestra.run.smithi005.stdout:90194313219 2024-09-15T03:40:50.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:40:50.096 INFO:teuthology.orchestra.run.smithi005.stdout:90194313220 2024-09-15T03:40:50.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:40:50.111 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:50.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:40:50.112 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.112 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-15T03:40:50.121 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.122 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:40:50.127 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.127 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-15T03:40:50.133 INFO:teuthology.orchestra.run.smithi005.stdout:94489280517 2024-09-15T03:40:50.133 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:40:50.226 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:40:50.869 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.7 2024-09-15T03:40:50.903 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:50.927 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:50.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.4 2024-09-15T03:40:51.024 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.5 2024-09-15T03:40:51.125 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.125 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:51.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.2 2024-09-15T03:40:51.165 INFO:teuthology.orchestra.run.smithi005.stdout:90194313221 2024-09-15T03:40:51.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.8 2024-09-15T03:40:51.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.15 2024-09-15T03:40:51.188 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:51.191 INFO:teuthology.orchestra.run.smithi005.stdout:90194313218 2024-09-15T03:40:51.213 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:51.228 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.229 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.229 INFO:teuthology.orchestra.run.smithi005.stdout:90194313219 2024-09-15T03:40:51.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.11 2024-09-15T03:40:51.250 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313218 for osd.6 2024-09-15T03:40:51.255 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:51.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.0 2024-09-15T03:40:51.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.9 2024-09-15T03:40:51.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.13 2024-09-15T03:40:51.261 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.265 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.10 2024-09-15T03:40:51.274 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:51.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.1 2024-09-15T03:40:51.280 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.12 2024-09-15T03:40:51.289 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:51.292 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.14 2024-09-15T03:40:51.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.3 2024-09-15T03:40:51.944 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:40:52.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:40:52.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:40:52.180 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:40:52.244 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:52.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:40:52.251 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:40:52.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:40:52.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.4 2024-09-15T03:40:52.261 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:40:52.261 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-15T03:40:52.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:40:52.278 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:40:52.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-15T03:40:52.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-15T03:40:52.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:40:52.427 INFO:teuthology.orchestra.run.smithi005.stdout:94489280514 2024-09-15T03:40:52.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.5 2024-09-15T03:40:52.716 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:52.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.2 2024-09-15T03:40:52.748 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:52.823 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:52.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2024-09-15T03:40:52.861 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:52.934 INFO:teuthology.orchestra.run.smithi005.stdout:90194313220 2024-09-15T03:40:52.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313220 for osd.6 2024-09-15T03:40:52.979 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.003 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:53.013 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:53.028 INFO:teuthology.orchestra.run.smithi005.stdout:94489280517 2024-09-15T03:40:53.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280516 for osd.11 2024-09-15T03:40:53.030 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.13 2024-09-15T03:40:53.038 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.043 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:53.045 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.9 2024-09-15T03:40:53.045 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.057 INFO:teuthology.orchestra.run.smithi005.stdout:90194313221 2024-09-15T03:40:53.061 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.1 2024-09-15T03:40:53.062 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.073 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313221 for osd.0 2024-09-15T03:40:53.073 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.092 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:53.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.10 2024-09-15T03:40:53.106 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.113 INFO:teuthology.orchestra.run.smithi005.stdout:94489280517 2024-09-15T03:40:53.113 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:53.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.3 2024-09-15T03:40:53.127 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.14 2024-09-15T03:40:53.128 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.155 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:53.168 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280516 for osd.12 2024-09-15T03:40:53.168 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:40:53.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:40:53.543 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-15T03:40:53.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2024-09-15T03:40:53.556 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.731 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-15T03:40:53.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.5 2024-09-15T03:40:53.744 DEBUG:teuthology.parallel:result is None 2024-09-15T03:40:53.744 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:40:53.744 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:40:54.020 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:40:54.032 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T03:40:54.014624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4763696,"kb_used_data":5948,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505185744,"statfs":{"total":1546188226560,"available":1541310201856,"internally_reserved":0,"allocated":6090752,"data_stored":3119300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313146+0000","last_change":"2024-09-15T03:40:50.313338+0000","last_active":"2024-09-15T03:40:50.313146+0000","last_peered":"2024-09-15T03:40:50.313146+0000","last_clean":"2024-09-15T03:40:50.313146+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:40:50.313146+0000","last_undegraded":"2024-09-15T03:40:50.313146+0000","last_fullsized":"2024-09-15T03:40:50.313146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:58.126050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.669764+0000","last_change":"2024-09-15T03:40:48.669967+0000","last_active":"2024-09-15T03:40:48.669764+0000","last_peered":"2024-09-15T03:40:48.669764+0000","last_clean":"2024-09-15T03:40:48.669764+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:40:48.669764+0000","last_undegraded":"2024-09-15T03:40:48.669764+0000","last_fullsized":"2024-09-15T03:40:48.669764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.746377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.042697+0000","last_change":"2024-09-15T03:40:49.042892+0000","last_active":"2024-09-15T03:40:49.042697+0000","last_peered":"2024-09-15T03:40:49.042697+0000","last_clean":"2024-09-15T03:40:49.042697+0000","last_became_active":"2024-09-15T03:40:46.656459+0000","last_became_peered":"2024-09-15T03:40:46.656459+0000","last_unstale":"2024-09-15T03:40:49.042697+0000","last_undegraded":"2024-09-15T03:40:49.042697+0000","last_fullsized":"2024-09-15T03:40:49.042697+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:52.988670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.005342+0000","last_change":"2024-09-15T03:40:49.005549+0000","last_active":"2024-09-15T03:40:49.005342+0000","last_peered":"2024-09-15T03:40:49.005342+0000","last_clean":"2024-09-15T03:40:49.005342+0000","last_became_active":"2024-09-15T03:40:46.669330+0000","last_became_peered":"2024-09-15T03:40:46.669330+0000","last_unstale":"2024-09-15T03:40:49.005342+0000","last_undegraded":"2024-09-15T03:40:49.005342+0000","last_fullsized":"2024-09-15T03:40:49.005342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:12.749616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000296755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.667305+0000","last_change":"2024-09-15T03:40:48.667305+0000","last_active":"2024-09-15T03:40:48.667305+0000","last_peered":"2024-09-15T03:40:48.667305+0000","last_clean":"2024-09-15T03:40:48.667305+0000","last_became_active":"2024-09-15T03:40:46.775427+0000","last_became_peered":"2024-09-15T03:40:46.775427+0000","last_unstale":"2024-09-15T03:40:48.667305+0000","last_undegraded":"2024-09-15T03:40:48.667305+0000","last_fullsized":"2024-09-15T03:40:48.667305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:00.077013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084937499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313237+0000","last_change":"2024-09-15T03:40:50.313452+0000","last_active":"2024-09-15T03:40:50.313237+0000","last_peered":"2024-09-15T03:40:50.313237+0000","last_clean":"2024-09-15T03:40:50.313237+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:40:50.313237+0000","last_undegraded":"2024-09-15T03:40:50.313237+0000","last_fullsized":"2024-09-15T03:40:50.313237+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.213415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.666602+0000","last_change":"2024-09-15T03:40:48.666791+0000","last_active":"2024-09-15T03:40:48.666602+0000","last_peered":"2024-09-15T03:40:48.666602+0000","last_clean":"2024-09-15T03:40:48.666602+0000","last_became_active":"2024-09-15T03:40:46.657361+0000","last_became_peered":"2024-09-15T03:40:46.657361+0000","last_unstale":"2024-09-15T03:40:48.666602+0000","last_undegraded":"2024-09-15T03:40:48.666602+0000","last_fullsized":"2024-09-15T03:40:48.666602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:06.411518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000334655,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.350423+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:40:50.350423+0000","last_peered":"2024-09-15T03:40:50.350423+0000","last_clean":"2024-09-15T03:40:50.350423+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:40:50.350423+0000","last_undegraded":"2024-09-15T03:40:50.350423+0000","last_fullsized":"2024-09-15T03:40:50.350423+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"24'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T03:40:47.838157+0000","last_change":"2024-09-15T03:40:43.639536+0000","last_active":"2024-09-15T03:40:47.838157+0000","last_peered":"2024-09-15T03:40:47.838157+0000","last_clean":"2024-09-15T03:40:47.838157+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:40:47.838157+0000","last_undegraded":"2024-09-15T03:40:47.838157+0000","last_fullsized":"2024-09-15T03:40:47.838157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:40:42.622204+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:49.505752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298188,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073652,"statfs":{"total":96636764160,"available":96331419648,"internally_reserved":0,"allocated":847872,"data_stored":658021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":839680,"data_stored":656829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":748.04600000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":319488,"data_stored":129112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":128534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297660,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074180,"statfs":{"total":96636764160,"available":96331960320,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:40:54.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:40:54.310 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:54.311 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:40:54.322 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T03:40:54.014624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4763696,"kb_used_data":5948,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505185744,"statfs":{"total":1546188226560,"available":1541310201856,"internally_reserved":0,"allocated":6090752,"data_stored":3119300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313146+0000","last_change":"2024-09-15T03:40:50.313338+0000","last_active":"2024-09-15T03:40:50.313146+0000","last_peered":"2024-09-15T03:40:50.313146+0000","last_clean":"2024-09-15T03:40:50.313146+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:40:50.313146+0000","last_undegraded":"2024-09-15T03:40:50.313146+0000","last_fullsized":"2024-09-15T03:40:50.313146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:58.126050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.669764+0000","last_change":"2024-09-15T03:40:48.669967+0000","last_active":"2024-09-15T03:40:48.669764+0000","last_peered":"2024-09-15T03:40:48.669764+0000","last_clean":"2024-09-15T03:40:48.669764+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:40:48.669764+0000","last_undegraded":"2024-09-15T03:40:48.669764+0000","last_fullsized":"2024-09-15T03:40:48.669764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.746377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.042697+0000","last_change":"2024-09-15T03:40:49.042892+0000","last_active":"2024-09-15T03:40:49.042697+0000","last_peered":"2024-09-15T03:40:49.042697+0000","last_clean":"2024-09-15T03:40:49.042697+0000","last_became_active":"2024-09-15T03:40:46.656459+0000","last_became_peered":"2024-09-15T03:40:46.656459+0000","last_unstale":"2024-09-15T03:40:49.042697+0000","last_undegraded":"2024-09-15T03:40:49.042697+0000","last_fullsized":"2024-09-15T03:40:49.042697+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:52.988670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.005342+0000","last_change":"2024-09-15T03:40:49.005549+0000","last_active":"2024-09-15T03:40:49.005342+0000","last_peered":"2024-09-15T03:40:49.005342+0000","last_clean":"2024-09-15T03:40:49.005342+0000","last_became_active":"2024-09-15T03:40:46.669330+0000","last_became_peered":"2024-09-15T03:40:46.669330+0000","last_unstale":"2024-09-15T03:40:49.005342+0000","last_undegraded":"2024-09-15T03:40:49.005342+0000","last_fullsized":"2024-09-15T03:40:49.005342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:12.749616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000296755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.667305+0000","last_change":"2024-09-15T03:40:48.667305+0000","last_active":"2024-09-15T03:40:48.667305+0000","last_peered":"2024-09-15T03:40:48.667305+0000","last_clean":"2024-09-15T03:40:48.667305+0000","last_became_active":"2024-09-15T03:40:46.775427+0000","last_became_peered":"2024-09-15T03:40:46.775427+0000","last_unstale":"2024-09-15T03:40:48.667305+0000","last_undegraded":"2024-09-15T03:40:48.667305+0000","last_fullsized":"2024-09-15T03:40:48.667305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:00.077013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084937499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313237+0000","last_change":"2024-09-15T03:40:50.313452+0000","last_active":"2024-09-15T03:40:50.313237+0000","last_peered":"2024-09-15T03:40:50.313237+0000","last_clean":"2024-09-15T03:40:50.313237+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:40:50.313237+0000","last_undegraded":"2024-09-15T03:40:50.313237+0000","last_fullsized":"2024-09-15T03:40:50.313237+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.213415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.666602+0000","last_change":"2024-09-15T03:40:48.666791+0000","last_active":"2024-09-15T03:40:48.666602+0000","last_peered":"2024-09-15T03:40:48.666602+0000","last_clean":"2024-09-15T03:40:48.666602+0000","last_became_active":"2024-09-15T03:40:46.657361+0000","last_became_peered":"2024-09-15T03:40:46.657361+0000","last_unstale":"2024-09-15T03:40:48.666602+0000","last_undegraded":"2024-09-15T03:40:48.666602+0000","last_fullsized":"2024-09-15T03:40:48.666602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:06.411518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000334655,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.350423+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:40:50.350423+0000","last_peered":"2024-09-15T03:40:50.350423+0000","last_clean":"2024-09-15T03:40:50.350423+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:40:50.350423+0000","last_undegraded":"2024-09-15T03:40:50.350423+0000","last_fullsized":"2024-09-15T03:40:50.350423+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"24'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T03:40:47.838157+0000","last_change":"2024-09-15T03:40:43.639536+0000","last_active":"2024-09-15T03:40:47.838157+0000","last_peered":"2024-09-15T03:40:47.838157+0000","last_clean":"2024-09-15T03:40:47.838157+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:40:47.838157+0000","last_undegraded":"2024-09-15T03:40:47.838157+0000","last_fullsized":"2024-09-15T03:40:47.838157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:40:42.622204+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:49.505752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298188,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073652,"statfs":{"total":96636764160,"available":96331419648,"internally_reserved":0,"allocated":847872,"data_stored":658021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":839680,"data_stored":656829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":748.04600000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":319488,"data_stored":129112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":128534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297660,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074180,"statfs":{"total":96636764160,"available":96331960320,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:40:54.324 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:40:54.324 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T03:40:54.325 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T03:40:54.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T03:40:54.640 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:54.640 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T03:40:54.652 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T03:40:54.653 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-15T03:40:54.663 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-15T03:40:54.664 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:40:54.664 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-15T03:40:54.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-15T03:40:54.961 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-15T03:40:54.961 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:40:54.961 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:40:55.240 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:40:55.252 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T03:40:54.014624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4763696,"kb_used_data":5948,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505185744,"statfs":{"total":1546188226560,"available":1541310201856,"internally_reserved":0,"allocated":6090752,"data_stored":3119300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313146+0000","last_change":"2024-09-15T03:40:50.313338+0000","last_active":"2024-09-15T03:40:50.313146+0000","last_peered":"2024-09-15T03:40:50.313146+0000","last_clean":"2024-09-15T03:40:50.313146+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:40:50.313146+0000","last_undegraded":"2024-09-15T03:40:50.313146+0000","last_fullsized":"2024-09-15T03:40:50.313146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:58.126050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.669764+0000","last_change":"2024-09-15T03:40:48.669967+0000","last_active":"2024-09-15T03:40:48.669764+0000","last_peered":"2024-09-15T03:40:48.669764+0000","last_clean":"2024-09-15T03:40:48.669764+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:40:48.669764+0000","last_undegraded":"2024-09-15T03:40:48.669764+0000","last_fullsized":"2024-09-15T03:40:48.669764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.746377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.042697+0000","last_change":"2024-09-15T03:40:49.042892+0000","last_active":"2024-09-15T03:40:49.042697+0000","last_peered":"2024-09-15T03:40:49.042697+0000","last_clean":"2024-09-15T03:40:49.042697+0000","last_became_active":"2024-09-15T03:40:46.656459+0000","last_became_peered":"2024-09-15T03:40:46.656459+0000","last_unstale":"2024-09-15T03:40:49.042697+0000","last_undegraded":"2024-09-15T03:40:49.042697+0000","last_fullsized":"2024-09-15T03:40:49.042697+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:52.988670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.005342+0000","last_change":"2024-09-15T03:40:49.005549+0000","last_active":"2024-09-15T03:40:49.005342+0000","last_peered":"2024-09-15T03:40:49.005342+0000","last_clean":"2024-09-15T03:40:49.005342+0000","last_became_active":"2024-09-15T03:40:46.669330+0000","last_became_peered":"2024-09-15T03:40:46.669330+0000","last_unstale":"2024-09-15T03:40:49.005342+0000","last_undegraded":"2024-09-15T03:40:49.005342+0000","last_fullsized":"2024-09-15T03:40:49.005342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:12.749616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000296755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.667305+0000","last_change":"2024-09-15T03:40:48.667305+0000","last_active":"2024-09-15T03:40:48.667305+0000","last_peered":"2024-09-15T03:40:48.667305+0000","last_clean":"2024-09-15T03:40:48.667305+0000","last_became_active":"2024-09-15T03:40:46.775427+0000","last_became_peered":"2024-09-15T03:40:46.775427+0000","last_unstale":"2024-09-15T03:40:48.667305+0000","last_undegraded":"2024-09-15T03:40:48.667305+0000","last_fullsized":"2024-09-15T03:40:48.667305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:00.077013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084937499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313237+0000","last_change":"2024-09-15T03:40:50.313452+0000","last_active":"2024-09-15T03:40:50.313237+0000","last_peered":"2024-09-15T03:40:50.313237+0000","last_clean":"2024-09-15T03:40:50.313237+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:40:50.313237+0000","last_undegraded":"2024-09-15T03:40:50.313237+0000","last_fullsized":"2024-09-15T03:40:50.313237+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.213415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.666602+0000","last_change":"2024-09-15T03:40:48.666791+0000","last_active":"2024-09-15T03:40:48.666602+0000","last_peered":"2024-09-15T03:40:48.666602+0000","last_clean":"2024-09-15T03:40:48.666602+0000","last_became_active":"2024-09-15T03:40:46.657361+0000","last_became_peered":"2024-09-15T03:40:46.657361+0000","last_unstale":"2024-09-15T03:40:48.666602+0000","last_undegraded":"2024-09-15T03:40:48.666602+0000","last_fullsized":"2024-09-15T03:40:48.666602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:06.411518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000334655,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.350423+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:40:50.350423+0000","last_peered":"2024-09-15T03:40:50.350423+0000","last_clean":"2024-09-15T03:40:50.350423+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:40:50.350423+0000","last_undegraded":"2024-09-15T03:40:50.350423+0000","last_fullsized":"2024-09-15T03:40:50.350423+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"24'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T03:40:47.838157+0000","last_change":"2024-09-15T03:40:43.639536+0000","last_active":"2024-09-15T03:40:47.838157+0000","last_peered":"2024-09-15T03:40:47.838157+0000","last_clean":"2024-09-15T03:40:47.838157+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:40:47.838157+0000","last_undegraded":"2024-09-15T03:40:47.838157+0000","last_fullsized":"2024-09-15T03:40:47.838157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:40:42.622204+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:49.505752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298188,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073652,"statfs":{"total":96636764160,"available":96331419648,"internally_reserved":0,"allocated":847872,"data_stored":658021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":839680,"data_stored":656829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":748.04600000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":319488,"data_stored":129112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":128534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297660,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074180,"statfs":{"total":96636764160,"available":96331960320,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:40:55.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:40:55.536 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:40:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:40:55.548 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T03:40:54.014624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4763696,"kb_used_data":5948,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505185744,"statfs":{"total":1546188226560,"available":1541310201856,"internally_reserved":0,"allocated":6090752,"data_stored":3119300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313146+0000","last_change":"2024-09-15T03:40:50.313338+0000","last_active":"2024-09-15T03:40:50.313146+0000","last_peered":"2024-09-15T03:40:50.313146+0000","last_clean":"2024-09-15T03:40:50.313146+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:40:50.313146+0000","last_undegraded":"2024-09-15T03:40:50.313146+0000","last_fullsized":"2024-09-15T03:40:50.313146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:58.126050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.669764+0000","last_change":"2024-09-15T03:40:48.669967+0000","last_active":"2024-09-15T03:40:48.669764+0000","last_peered":"2024-09-15T03:40:48.669764+0000","last_clean":"2024-09-15T03:40:48.669764+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:40:48.669764+0000","last_undegraded":"2024-09-15T03:40:48.669764+0000","last_fullsized":"2024-09-15T03:40:48.669764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.746377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.042697+0000","last_change":"2024-09-15T03:40:49.042892+0000","last_active":"2024-09-15T03:40:49.042697+0000","last_peered":"2024-09-15T03:40:49.042697+0000","last_clean":"2024-09-15T03:40:49.042697+0000","last_became_active":"2024-09-15T03:40:46.656459+0000","last_became_peered":"2024-09-15T03:40:46.656459+0000","last_unstale":"2024-09-15T03:40:49.042697+0000","last_undegraded":"2024-09-15T03:40:49.042697+0000","last_fullsized":"2024-09-15T03:40:49.042697+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:52.988670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:49.005342+0000","last_change":"2024-09-15T03:40:49.005549+0000","last_active":"2024-09-15T03:40:49.005342+0000","last_peered":"2024-09-15T03:40:49.005342+0000","last_clean":"2024-09-15T03:40:49.005342+0000","last_became_active":"2024-09-15T03:40:46.669330+0000","last_became_peered":"2024-09-15T03:40:46.669330+0000","last_unstale":"2024-09-15T03:40:49.005342+0000","last_undegraded":"2024-09-15T03:40:49.005342+0000","last_fullsized":"2024-09-15T03:40:49.005342+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:12.749616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000296755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.667305+0000","last_change":"2024-09-15T03:40:48.667305+0000","last_active":"2024-09-15T03:40:48.667305+0000","last_peered":"2024-09-15T03:40:48.667305+0000","last_clean":"2024-09-15T03:40:48.667305+0000","last_became_active":"2024-09-15T03:40:46.775427+0000","last_became_peered":"2024-09-15T03:40:46.775427+0000","last_unstale":"2024-09-15T03:40:48.667305+0000","last_undegraded":"2024-09-15T03:40:48.667305+0000","last_fullsized":"2024-09-15T03:40:48.667305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:00.077013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084937499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.313237+0000","last_change":"2024-09-15T03:40:50.313452+0000","last_active":"2024-09-15T03:40:50.313237+0000","last_peered":"2024-09-15T03:40:50.313237+0000","last_clean":"2024-09-15T03:40:50.313237+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:40:50.313237+0000","last_undegraded":"2024-09-15T03:40:50.313237+0000","last_fullsized":"2024-09-15T03:40:50.313237+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.213415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:48.666602+0000","last_change":"2024-09-15T03:40:48.666791+0000","last_active":"2024-09-15T03:40:48.666602+0000","last_peered":"2024-09-15T03:40:48.666602+0000","last_clean":"2024-09-15T03:40:48.666602+0000","last_became_active":"2024-09-15T03:40:46.657361+0000","last_became_peered":"2024-09-15T03:40:46.657361+0000","last_unstale":"2024-09-15T03:40:48.666602+0000","last_undegraded":"2024-09-15T03:40:48.666602+0000","last_fullsized":"2024-09-15T03:40:48.666602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:06.411518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000334655,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-15T03:40:50.350423+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:40:50.350423+0000","last_peered":"2024-09-15T03:40:50.350423+0000","last_clean":"2024-09-15T03:40:50.350423+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:40:50.350423+0000","last_undegraded":"2024-09-15T03:40:50.350423+0000","last_fullsized":"2024-09-15T03:40:50.350423+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"24'112","reported_seq":149,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-15T03:40:47.838157+0000","last_change":"2024-09-15T03:40:43.639536+0000","last_active":"2024-09-15T03:40:47.838157+0000","last_peered":"2024-09-15T03:40:47.838157+0000","last_clean":"2024-09-15T03:40:47.838157+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:40:47.838157+0000","last_undegraded":"2024-09-15T03:40:47.838157+0000","last_fullsized":"2024-09-15T03:40:47.838157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:40:42.622204+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:49.505752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298188,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073652,"statfs":{"total":96636764160,"available":96331419648,"internally_reserved":0,"allocated":847872,"data_stored":658021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":839680,"data_stored":656829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":748.04600000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]}]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":319488,"data_stored":129112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":315392,"data_stored":128534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297660,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074180,"statfs":{"total":96636764160,"available":96331960320,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,5,6,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297664,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074176,"statfs":{"total":96636764160,"available":96331956224,"internally_reserved":0,"allocated":311296,"data_stored":128515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":129093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:40:55.550 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:40:55.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-15T03:40:55.828 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 29 2024-09-15T03:40:55.828 INFO:teuthology.orchestra.run.smithi005.stdout:fsid da8e668f-207e-4717-9e66-7070b1986601 2024-09-15T03:40:55.828 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-15T03:40:30.211709+0000 2024-09-15T03:40:55.828 INFO:teuthology.orchestra.run.smithi005.stdout:modified 2024-09-15T03:40:48.655606+0000 2024-09-15T03:40:55.828 INFO:teuthology.orchestra.run.smithi005.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-15T03:40:55.829 INFO:teuthology.orchestra.run.smithi005.stdout:crush_version 5 2024-09-15T03:40:55.829 INFO:teuthology.orchestra.run.smithi005.stdout:full_ratio 0.95 2024-09-15T03:40:55.829 INFO:teuthology.orchestra.run.smithi005.stdout:backfillfull_ratio 0.9 2024-09-15T03:40:55.829 INFO:teuthology.orchestra.run.smithi005.stdout:nearfull_ratio 0.85 2024-09-15T03:40:55.829 INFO:teuthology.orchestra.run.smithi005.stdout:require_min_compat_client luminous 2024-09-15T03:40:55.830 INFO:teuthology.orchestra.run.smithi005.stdout:min_compat_client jewel 2024-09-15T03:40:55.830 INFO:teuthology.orchestra.run.smithi005.stdout:require_osd_release quincy 2024-09-15T03:40:55.830 INFO:teuthology.orchestra.run.smithi005.stdout:stretch_mode_enabled false 2024-09-15T03:40:55.830 INFO:teuthology.orchestra.run.smithi005.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-15T03:40:55.830 INFO:teuthology.orchestra.run.smithi005.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 29 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: removed_snaps_queue [2~1] 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout:max_osd 16 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6800/33102,v1:172.21.15.5:6801/33102] [v2:172.21.15.5:6802/33102,v1:172.21.15.5:6803/33102] exists,up 2efef372-d786-40e5-96a1-10f399314942 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6810/33259,v1:172.21.15.64:6812/33259] [v2:172.21.15.64:6814/33259,v1:172.21.15.64:6816/33259] exists,up d77b5a7a-5228-4bed-b89b-799b0d0b9491 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6809/32810,v1:172.21.15.78:6811/32810] [v2:172.21.15.78:6813/32810,v1:172.21.15.78:6815/32810] exists,up e81da5de-c743-4cf0-a36a-beee634f01da 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6802/33113,v1:172.21.15.105:6803/33113] [v2:172.21.15.105:6804/33113,v1:172.21.15.105:6805/33113] exists,up dae3f484-9f24-4f24-96c3-253465d0d369 2024-09-15T03:40:55.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6816/33104,v1:172.21.15.5:6817/33104] [v2:172.21.15.5:6818/33104,v1:172.21.15.5:6819/33104] exists,up 7a2d4b3d-e398-4010-8952-4e5207e005b2 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/33258,v1:172.21.15.64:6801/33258] [v2:172.21.15.64:6802/33258,v1:172.21.15.64:6803/33258] exists,up b757a5a4-3278-42ec-b43f-6c08bfc15f06 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6800/32809,v1:172.21.15.78:6801/32809] [v2:172.21.15.78:6802/32809,v1:172.21.15.78:6803/32809] exists,up 96572944-7f1f-4461-a818-89bd0c4e2faf 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6812/33114,v1:172.21.15.105:6814/33114] [v2:172.21.15.105:6816/33114,v1:172.21.15.105:6819/33114] exists,up e698cf1c-5b5d-43a2-b44a-3d30af6f5917 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6808/33107,v1:172.21.15.5:6809/33107] [v2:172.21.15.5:6810/33107,v1:172.21.15.5:6811/33107] exists,up 12a65fa6-1fbd-4900-88c9-69695141bc1b 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/33260,v1:172.21.15.64:6809/33260] [v2:172.21.15.64:6811/33260,v1:172.21.15.64:6813/33260] exists,up 8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6824/32811,v1:172.21.15.78:6825/32811] [v2:172.21.15.78:6826/32811,v1:172.21.15.78:6827/32811] exists,up 91199e3c-694d-4888-859b-b76feac7c96d 2024-09-15T03:40:55.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6810/33115,v1:172.21.15.105:6811/33115] [v2:172.21.15.105:6813/33115,v1:172.21.15.105:6815/33115] exists,up 4d6ca3aa-f880-4673-8d66-21d438dd4b61 2024-09-15T03:40:55.833 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6824/33108,v1:172.21.15.5:6825/33108] [v2:172.21.15.5:6826/33108,v1:172.21.15.5:6827/33108] exists,up 5d4556fd-7659-4e2b-b60b-e44eca29c3e6 2024-09-15T03:40:55.833 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] [v2:172.21.15.64:6826/33261,v1:172.21.15.64:6827/33261] exists,up 78791966-b232-400f-9c30-cabebfa73dc0 2024-09-15T03:40:55.833 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] [v2:172.21.15.78:6812/32812,v1:172.21.15.78:6814/32812] exists,up 4a275057-7e86-452f-958d-80263088db83 2024-09-15T03:40:55.833 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] [v2:172.21.15.105:6828/33116,v1:172.21.15.105:6829/33116] exists,up 18589d26-e121-4e72-bf6e-24a4a37c49c5 2024-09-15T03:40:55.840 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6800/33102,v1:172.21.15.5:6801/33102] [v2:172.21.15.5:6802/33102,v1:172.21.15.5:6803/33102] exists,up 2efef372-d786-40e5-96a1-10f399314942', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6810/33259,v1:172.21.15.64:6812/33259] [v2:172.21.15.64:6814/33259,v1:172.21.15.64:6816/33259] exists,up d77b5a7a-5228-4bed-b89b-799b0d0b9491', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6809/32810,v1:172.21.15.78:6811/32810] [v2:172.21.15.78:6813/32810,v1:172.21.15.78:6815/32810] exists,up e81da5de-c743-4cf0-a36a-beee634f01da', 'osd.3 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6802/33113,v1:172.21.15.105:6803/33113] [v2:172.21.15.105:6804/33113,v1:172.21.15.105:6805/33113] exists,up dae3f484-9f24-4f24-96c3-253465d0d369', 'osd.4 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6816/33104,v1:172.21.15.5:6817/33104] [v2:172.21.15.5:6818/33104,v1:172.21.15.5:6819/33104] exists,up 7a2d4b3d-e398-4010-8952-4e5207e005b2', 'osd.5 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/33258,v1:172.21.15.64:6801/33258] [v2:172.21.15.64:6802/33258,v1:172.21.15.64:6803/33258] exists,up b757a5a4-3278-42ec-b43f-6c08bfc15f06', 'osd.6 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6800/32809,v1:172.21.15.78:6801/32809] [v2:172.21.15.78:6802/32809,v1:172.21.15.78:6803/32809] exists,up 96572944-7f1f-4461-a818-89bd0c4e2faf', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6812/33114,v1:172.21.15.105:6814/33114] [v2:172.21.15.105:6816/33114,v1:172.21.15.105:6819/33114] exists,up e698cf1c-5b5d-43a2-b44a-3d30af6f5917', 'osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6808/33107,v1:172.21.15.5:6809/33107] [v2:172.21.15.5:6810/33107,v1:172.21.15.5:6811/33107] exists,up 12a65fa6-1fbd-4900-88c9-69695141bc1b', 'osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/33260,v1:172.21.15.64:6809/33260] [v2:172.21.15.64:6811/33260,v1:172.21.15.64:6813/33260] exists,up 8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6824/32811,v1:172.21.15.78:6825/32811] [v2:172.21.15.78:6826/32811,v1:172.21.15.78:6827/32811] exists,up 91199e3c-694d-4888-859b-b76feac7c96d', 'osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6810/33115,v1:172.21.15.105:6811/33115] [v2:172.21.15.105:6813/33115,v1:172.21.15.105:6815/33115] exists,up 4d6ca3aa-f880-4673-8d66-21d438dd4b61', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.5:6824/33108,v1:172.21.15.5:6825/33108] [v2:172.21.15.5:6826/33108,v1:172.21.15.5:6827/33108] exists,up 5d4556fd-7659-4e2b-b60b-e44eca29c3e6', 'osd.13 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] [v2:172.21.15.64:6826/33261,v1:172.21.15.64:6827/33261] exists,up 78791966-b232-400f-9c30-cabebfa73dc0', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] [v2:172.21.15.78:6812/32812,v1:172.21.15.78:6814/32812] exists,up 4a275057-7e86-452f-958d-80263088db83', 'osd.15 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] [v2:172.21.15.105:6828/33116,v1:172.21.15.105:6829/33116] exists,up 18589d26-e121-4e72-bf6e-24a4a37c49c5'] 2024-09-15T03:40:55.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "mon.a", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster": "ceph", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "admin_socket_mode": "", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_client_required": "cephx, none", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_cluster_required": "cephx", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_debug": "true", 2024-09-15T03:40:55.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_service_required": "cephx", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_supported": "cephx", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio": "true", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_poll_ms": "250", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_reap_max": "16", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_async_discard": "false", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_block_size": "4096", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio": "false", 2024-09-15T03:40:55.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_inflight_ios": "false", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_enable_discard": "false", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_flock_retry": "3", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_inject_crash": "0", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring": "false", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring_hipri": "false", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-15T03:40:55.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_type": "", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_alloc_size": "1048576", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_allocator": "hybrid", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_buffered_io": "true", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_for_zeros": "false", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_compact_log_sync": "false", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-15T03:40:55.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_max_log_runway": "4194304", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_max_prefetch": "1048576", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_min_flush_size": "524288", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_min_log_runway": "1048576", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_replay_recovery": "false", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_sync_write": "false", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_allocation_from_file": "true", 2024-09-15T03:40:55.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_allocator": "hybrid", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_async_db_compaction": "true", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_create": "true", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_create": "false", 2024-09-15T03:40:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_path": "", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_size": "0", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_path": "", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_preallocate_file": "false", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_size": "107374182400", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_create": "false", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_path": "", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_size": "100663296", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs": "true", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-15T03:40:55.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_autotune": "true", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size": "0", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-15T03:40:55.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_type": "2q", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_clone_cow": "true", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_mode": "none", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-15T03:40:55.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_csum_type": "crc32c", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_freelist": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-15T03:40:55.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_small_allocations": "0", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_default_buffered_read": "true", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_default_buffered_write": "false", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-15T03:40:55.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fail_eio": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mount": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-15T03:40:55.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_umount": "false", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_ignore_data_csum": "false", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-15T03:40:55.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_op_age": "5.000000", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_alloc_size": "0", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size": "0", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_deferred_txc": "32", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size": "0", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-15T03:40:55.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_nid_prealloc": "1024", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_retry_disk_reads": "3", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_cf": "true", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-15T03:40:55.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_mem": "512", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_tracing": "false", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-15T03:40:55.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_zero_block_detection": "false", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-15T03:40:55.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_cluster_require_signatures": "false", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_cluster_require_version": "2", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_require_signatures": "false", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_require_version": "2", 2024-09-15T03:40:55.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_service_require_signatures": "false", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_service_require_version": "2", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_sign_messages": "true", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "chdir": "", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_acl_type": "", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_asio_thread_count": "2", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_cache_mid": "0.750000", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_cache_size": "16384", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_caps_release_delay": "5", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_check_pool_perm": "true", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-15T03:40:55.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_force_sync_read": "false", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_getattr_caps": "false", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_inject_features": "", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_inject_tick_delay": "0", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_die_on_failed_remount": "false", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_dirsize_rbytes": "true", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_force_lazyio": "false", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_fs": "", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_inject_release_failure": "false", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_max_inline_size": "4096", 2024-09-15T03:40:55.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mds_namespace": "", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_metadata": "", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_gid": "-1", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_timeout": "300", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_uid": "-1", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mountpoint": "/", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_notify_timeout": "10", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc": "true", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_dirty": "104857600", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_objects": "1000", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_size": "209715200", 2024-09-15T03:40:55.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_target_dirty": "8388608", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_permissions": "true", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_quota": "true", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_quota_df": "true", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_max_bytes": "0", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_max_periods": "4", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_min": "131072", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_reconnect_stale": "false", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_shutdown_timeout": "30", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_snapdir": ".snap", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_tick_interval": "1", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_trace": "", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_try_dentry_invalidate": "false", 2024-09-15T03:40:55.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_use_faked_inos": "false", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "client_use_random_mds": "false", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog": "false", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog_port": "12201", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_monitors": "default=true", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog": "false", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog_level": "info", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "-", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_network": "", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_network_interface": "", 2024-09-15T03:40:55.961 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_isal": "false", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_level": "5", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_winsize": "-15", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zstd_level": "1", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_alien_op_num_threads": "6", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location_hook": "", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location_hook_timeout": "10", 2024-09-15T03:40:55.962 INFO:teuthology.orchestra.run.smithi005.stdout: "daemonize": "false", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_alienstore": "0/5", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_allow_any_pool_priority": "false", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asok": "1/5", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asok_assert_abort": "false", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asserts_on_shutdown": "true", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_auth": "20/20", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bdev": "1/3", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bluefs": "1/5", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bluestore": "1/5", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_buffer": "0/1", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ceph_exporter": "1/5", 2024-09-15T03:40:55.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cephfs_mirror": "0/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cephsqlite": "0/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_civetweb": "1/10", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_client": "0/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_compressor": "1/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_context": "0/1", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crush": "1/1", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crypto": "1/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_deliberately_leak_memory": "false", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_disable_randomized_ping": "false", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_dpdk": "1/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_eventtrace": "1/5", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_filer": "0/1", 2024-09-15T03:40:55.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_filestore": "1/3", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_finisher": "1/1", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_fuse": "1/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_heartbeat_testing_span": "0", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_heartbeatmap": "1/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_javaclient": "1/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_journal": "1/3", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_journaler": "0/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_kstore": "1/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_leveldb": "4/5", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_lockdep": "0/1", 2024-09-15T03:40:55.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mclock": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_balancer": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_locker": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_log": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_log_expire": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_migrator": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_memdb": "4/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mgr": "2/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mgrc": "1/5", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mon": "20/20", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_monc": "0/10", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ms": "1/1", 2024-09-15T03:40:55.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_none": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objclass": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objectcacher": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objecter": "0/1", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_optracker": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_osd": "1/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_paxos": "20/20", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_perfcounter": "1/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_prioritycache": "1/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rados": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_mirror": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_pwl": "0/5", 2024-09-15T03:40:55.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_replay": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_refs": "0/0", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_reserver": "1/1", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw": "1/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_datacache": "1/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_sync": "1/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rocksdb": "4/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_cache": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_cleaner": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_device": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_journal": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_lba": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_odata": "0/5", 2024-09-15T03:40:55.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_omap": "0/5", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_onode": "0/5", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_tm": "0/5", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_striper": "0/1", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_test": "0/5", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_throttle": "1/1", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_timer": "0/1", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_tp": "0/5", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "device_failure_prediction_mode": "none", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_graylog": "false", 2024-09-15T03:40:55.969 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_journald": "false", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_stderr": "true", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_syslog": "false", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "event_tracing": "false", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_addr": "0.0.0.0", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_http_port": "9926", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_prio_limit": "5", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_sort_metrics": "true", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_stats_period": "5", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "fake_statfs_for_testing": "0", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "fatal_signal_handlers": "true", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "filer_max_purge_ops": "10", 2024-09-15T03:40:55.970 INFO:teuthology.orchestra.run.smithi005.stdout: "filer_max_truncate_ops": "128", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_apply_finisher_threads": "1", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_blackhole": "false", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_btrfs_clone_range": "true", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_btrfs_snap": "true", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_caller_concurrency": "10", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_collect_device_partition_information": "true", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_commit_timeout": "600.000000", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_inject_read_err": "false", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_omap_check": "false", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_verify_split": "false", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_dump_file": "", 2024-09-15T03:40:55.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fadvise": "true", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fail_eio": "true", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fd_cache_shards": "16", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fd_cache_size": "128", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fiemap": "false", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fiemap_threshold": "4096", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_inject_stall": "0", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_parallel": "false", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_trailing": "false", 2024-09-15T03:40:55.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_writeahead": "false", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_kill_at": "0", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs": "0", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-15T03:40:55.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size": "0", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_merge_threshold": "-10", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_odsync_write": "false", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_backend": "rocksdb", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_backend_path": "", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_thread_timeout": "60", 2024-09-15T03:40:55.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_threads": "2", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_punch_hole": "false", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_ops": "50", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-15T03:40:55.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_seek_data_hole": "false", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_sloppy_crc": "false", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_splice": "false", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_split_multiple": "2", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_split_rand_factor": "20", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_update_to": "1000", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-15T03:40:55.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_enable": "true", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_xfs_extsize": "false", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_zfs_snap": "false", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "fio_dir": "/tmp/fio", 2024-09-15T03:40:55.977 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "da8e668f-207e-4717-9e66-7070b1986601", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_allow_other": "true", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_atomic_o_trunc": "true", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_big_writes": "true", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_debug": "false", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_default_permissions": "false", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_disable_pagecache": "false", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_max_write": "0", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_multithreaded": "true", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_require_active_mds": "true", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_set_user_groups": "true", 2024-09-15T03:40:55.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_move": "true", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_read": "true", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_write": "true", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_use_invalidate_cb": "true", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "gss_target_name": "ceph", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_file": "", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_inject_failure": "0", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_interval": "5", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "host": "smithi005", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-15T03:40:55.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-15T03:40:55.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "inject_early_sigterm": "false", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "jaeger_tracing_enable": "false", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_aio": "true", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_align_min_size": "65536", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_block_align": "true", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_block_size": "4096", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_dio": "true", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_discard": "false", 2024-09-15T03:40:55.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_force_aio": "false", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_ignore_corruption": "false", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_max_write_bytes": "10485760", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_max_write_entries": "100", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_replay_from": "0", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_write_header_frequency": "0", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_zero_on_create": "false", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_prefetch_periods": "10", 2024-09-15T03:40:55.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_prezero_periods": "5", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_write_head_interval": "15", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "keyfile": "", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_backend": "rocksdb", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_default_stripe_size": "65536", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_fsck_on_mount": "false", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_max_bytes": "67108864", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_max_ops": "512", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_nid_prealloc": "1024", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_onode_map_size": "1024", 2024-09-15T03:40:55.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_sync_submit_transaction": "false", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_sync_transaction": "false", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_block_size": "65536", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_bloom_size": "0", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_cache_size": "536870912", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_compact_on_mount": "false", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_compression": "false", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_log": "", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_max_open_files": "0", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_paranoid": "false", 2024-09-15T03:40:55.984 INFO:teuthology.orchestra.run.smithi005.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "librados_thread_count": "2", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "lockdep": "false", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "lockdep_force_backtrace": "false", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_coarse_timestamps": "true", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_flush_on_exit": "false", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_graylog_host": "127.0.0.1", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_graylog_port": "12201", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_max_new": "1000", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_max_recent": "10000", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_stderr_prefix": "", 2024-09-15T03:40:55.985 INFO:teuthology.orchestra.run.smithi005.stdout: "log_stop_at_utilization": "0.970000", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_file": "true", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_graylog": "false", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_journald": "false", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_stderr": "false", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_syslog": "false", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "max_rotating_auth_attempts": "10", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_action_on_write_error": "1", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_alternate_name_max": "8192", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_asio_thread_count": "2", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_export_pin": "true", 2024-09-15T03:40:55.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_dirs": "true", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_interval": "5", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_interval": "10", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_max": "-1", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_max_until": "-1", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_merge_size": "50", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_midchunk": "0.300000", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_min_start": "0.200000", 2024-09-15T03:40:55.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_minchunk": "0.001000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_mode": "0", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_need_max": "1.200000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_need_min": "0.800000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_bits": "3", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_size": "10000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_target_decay": "10.000000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-15T03:40:55.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_grace": "15.000000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_interval": "4.000000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_mid": "0.700000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_release_free_interval": "10", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_reservation": "0.050000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_interval": "1", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_threshold": "262144", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-15T03:40:55.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_prealloc_inos": "1000", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_connect_bootstrapping": "false", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_damage_table_max_entries": "10000", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_auth_pins": "false", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_frag": "false", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_scatterstat": "false", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_subtrees": "false", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_decay_halflife": "5.000000", 2024-09-15T03:40:55.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_default_dir_hash": "2", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_defer_session_stale": "true", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_deny_all_reconnect": "false", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_keys_per_op": "16384", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_max_commit_size": "10", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_max_entries": "0", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_on_map": "false", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-15T03:40:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_early_reply": "true", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_enable_op_tracker": "true", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_enforce_unique_name": "true", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_random": "true", 2024-09-15T03:40:55.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_health_summarize_threshold": "10", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_health_dummy": "false", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_migrator_session_race": "false", 2024-09-15T03:40:55.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_join_fs": "", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_journal_format": "1", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_create_at": "0", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_export_at": "0", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_import_at": "0", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_at": "0", 2024-09-15T03:40:55.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_expire_at": "0", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_replay_at": "0", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_link_at": "0", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_mdstable_at": "0", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_openc_at": "0", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_rename_at": "0", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_events_per_segment": "1024", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_max_events": "-1", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_max_segments": "128", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_pause": "false", 2024-09-15T03:40:55.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_segment_size": "0", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_warn_factor": "2.000000", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_caps_per_client": "1048576", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_completed_flushes": "100000", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_completed_requests": "100000", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_export_size": "20971520", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_file_recover": "32", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_files": "64", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_ops": "8192", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-15T03:40:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_snaps_per_dir": "100", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_metrics_update_interval": "2", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_min_caps_per_client": "100", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_min_caps_working_set": "10000", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_numa_node": "-1", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_complaint_time": "30.000000", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_duration": "600", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_size": "20", 2024-09-15T03:40:55.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_slow_op_size": "20", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_log_threshold": "5", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_ping_grace": "15", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_ping_interval": "5", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_caps": "30000", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-15T03:40:55.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_warning_threshold": "262144", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_replay_interval": "1.000000", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_root_ino_gid": "0", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_root_ino_uid": "0", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-15T03:40:55.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_shutdown_check": "0", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_skip_ino": "0", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_max_uid": "4294967294", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_min_uid": "0", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_rstat": "false", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_standby_replay_damaged": "false", 2024-09-15T03:40:56.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_symlink_recovery": "true", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_thrash_exports": "0", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_thrash_fragments": "0", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_tick_interval": "5.000000", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_valgrind_exit": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_verify_backtrace": "1", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_verify_scatter": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_wipe_sessions": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mempool_debug": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_device_bytes": "1073741824", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_page_set": "false", 2024-09-15T03:40:56.001 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_page_size": "65536", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_bytes": "134217728", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_messages": "512", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_max_pg_num_change": "128", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mds_bytes": "134217728", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mds_messages": "128", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-15T03:40:56.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mon_bytes": "134217728", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mon_messages": "128", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_osd_bytes": "536870912", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_osd_messages": "8192", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_pool": "true", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_standby_modules": "true", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_stats_period": "5", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_stats_threshold": "5", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_tick_period": "2", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-15T03:40:56.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_allow_pool_delete": "true", 2024-09-15T03:40:56.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_allow_pool_size_one": "true", 2024-09-15T03:40:56.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_auth_validate_all_caps": "true", 2024-09-15T03:40:56.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-15T03:40:56.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-15T03:40:56.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_bytes": "104857600", 2024-09-15T03:40:56.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_directed_command_retry": "5", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_parallel": "3", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_log_interval": "1.000000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_ping_interval": "10.000000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-15T03:40:56.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_level": "debug", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_file": "true", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_journald": "false", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-15T03:40:56.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_bootstrap": "false", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_start": "false", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_trim": "true", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cpu_threads": "4", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_crush_min_required_version": "hammer", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_daemon_bytes": "419430400", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_avail_crit": "5", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_avail_warn": "5", 2024-09-15T03:40:56.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_size_warn": "16106127360", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_json": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_transactions": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_extra_checks": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_pacific": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_quincy": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-15T03:40:56.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_down_mkfs_grace": "120", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_election_default_strategy": "3", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_election_timeout": "5.000000", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ping_divisor": "2", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_enable_op_tracker": "true", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_fake_pool_delete": "false", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_force_quorum_join": "false", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_globalid_prealloc": "10000", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_detail_to_clog": "false", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_log_update_period": "5", 2024-09-15T03:40:56.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_max_detail": "50", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog": "true", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog_interval": "600", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_host": "172.21.15.5,172.21.15.64,172.21.15.78", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_host_override": "", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_initial_members": "", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease": "5.000000", 2024-09-15T03:40:56.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_full_interval": "50", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_max": "10000", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_max_summary": "50", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_log_epochs": "500", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_osd": "10000", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_pg_per_osd": "10000", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_pool_pg_num": "65536", 2024-09-15T03:40:56.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_force_trim_to": "0", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_skip_sanity": "false", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_memory_autotune": "true", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_memory_target": "2147483648", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_beacon_grace": "30", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_digest_period": "5", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_inactive_grace": "60", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-15T03:40:56.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_min_osdmap_epochs": "500", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_complaint_time": "30", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_duration": "600", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_size": "20", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_slow_op_size": "20", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_log_threshold": "5", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_in": "false", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-15T03:40:56.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_cache_size": "500", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_down_out_interval": "600", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_force_trim_to": "0", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-15T03:40:56.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_down_reporters": "2", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_report_timeout": "900", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-15T03:40:56.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_stuck_threshold": "60", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-15T03:40:56.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_probe_timeout": "2.000000", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_max_change": "0.050000", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_max_osds": "4", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-15T03:40:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_interval": "300", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_max_keys": "100", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_timeout": "300", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_session_timeout": "300", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_smart_report_timeout": "5", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stat_smooth_intervals": "6", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_pool_min_size": "2", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_pool_size": "4", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_debug": "false", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-15T03:40:56.018 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_provider_kill_at": "0", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_requester_kill_at": "0", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_timeout": "60.000000", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_target_pg_per_osd": "100", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_tick_interval": "5", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_timecheck_interval": "300.000000", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_older_version_delay": "604800", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-15T03:40:56.019 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_misplaced": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_older_version": "true", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-15T03:40:56.020 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": "", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_op_threads": "3", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_cm": "false", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_device_name": "", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_dscp": "96", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_local_gid": "", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-15T03:40:56.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_port_num": "1", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_sl": "3", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_support_srq": "true", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_type": "ib", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_reap_threshold": "5", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_before_connect": "false", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_ipv4": "true", 2024-09-15T03:40:56.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_ipv6": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_msgr1": "true", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_msgr2": "true", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_port_max": "7568", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_port_min": "6800", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_retry_count": "3", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_retry_delay": "5", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_client": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mds": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mgr": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mon": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_osd": "false", 2024-09-15T03:40:56.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_client_mode": "crc secure", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_cluster_mode": "crc secure", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_cluster_type": "", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_compress_secure": "false", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_connection_idle_timeout": "900", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_connection_ready_timeout": "10", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_crc_data": "true", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_crc_header": "true", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_bad_msg": "false", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_bug": "true", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_old_message": "true", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_skipped_message": "false", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-15T03:40:56.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_coremask": "0xF", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_enable_tso": "true", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hugepages": "", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_lro": "true", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_memory_channel": "4", 2024-09-15T03:40:56.025 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_pmd": "", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_port_id": "0", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dump_on_send": "false", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_initial_backoff": "0.200000", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_max": "1.000000", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_type": "osd", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_socket_failures": "2500", 2024-09-15T03:40:56.026 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_learn_addr_from_peer": "true", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_max_accept_failures": "4", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_max_backoff": "15.000000", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_client_mode": "secure crc", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_service_mode": "secure crc", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compress_min_size": "1024", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compress_mode": "none", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_pq_min_cost": "65536", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_public_type": "", 2024-09-15T03:40:56.027 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_service_mode": "crc secure", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_listen_backlog": "512", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_nodelay": "true", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_rcvbuf": "0", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_type": "async+posix", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "no_config_file": "false", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_completion_locks_per_session": "32", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inflight_ops": "1024", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-15T03:40:56.028 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_tick_interval": "5.000000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_timeout": "10.000000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "objectstore_blackhole": "false", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "openssl_engine_opts": "", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_delay_time": "5.000000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_hist_halflife": "1000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_max_low_ops": "2", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_max_ops": "4", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_slop": "0.020000", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-15T03:40:56.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_asio_thread_count": "2", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_async_recovery_min_cost": "1", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_scan_max": "512", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_scan_min": "64", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_degraded": "false", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_peering": "false", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_unfound": "true", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_beacon_report_interval": "300", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_duration": "30", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_max_block_size": "67108864", 2024-09-15T03:40:56.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_blkin_trace_all": "false", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_check_for_log_corruption": "false", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_update_on_start": "true", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_message_cap": "256", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_message_size_cap": "524288000", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_op_priority": "63", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_watch_timeout": "30", 2024-09-15T03:40:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_command_max_records": "256", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_compact_on_start": "false", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_update_on_start": "true", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_update_weight_set": "true", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_feed_pullee": "-1", 2024-09-15T03:40:56.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_misdirected_ops": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_no_acting_change": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_no_purge_strays": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_op_order": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_shutdown": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-15T03:40:56.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_snaps": "false", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_keys": "1024", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_stride": "524288", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_default_notify_timeout": "30", 2024-09-15T03:40:56.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep": "0.000000", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_discard_disconnected_ops": "true", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_enable_op_tracker": "true", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_info": "true", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown": "true", 2024-09-15T03:40:56.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_function_tracing": "false", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_grace": "20", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_interval": "6", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_peers": "10", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_size": "2000", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_stale": "600", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-15T03:40:56.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_max_size": "100000", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_min_size": "1000", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal_size": "5120", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_kill_backfill_at": "0", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_cache_size": "50", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_dedup": "true", 2024-09-15T03:40:56.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_message_max": "40", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_share_max_epochs": "40", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_attr_name_len": "100", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_attr_size": "0", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_backfills": "1", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_markdown_count": "5", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_markdown_period": "600", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_name_len": "2048", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_namespace_len": "256", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_size": "134217728", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-15T03:40:56.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_blocked_by": "16", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_log_entries": "10000", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pgls": "1024", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_push_cost": "8388608", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_push_objects": "10", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_scrubs": "3", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_trimming_pgs": "2", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_write_op_reply_len": "64", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_write_size": "90", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-15T03:40:56.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_profile": "balanced", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-15T03:40:56.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_base": "805306368", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_cache_min": "134217728", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target": "4294967296", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target_autotune": "false", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_min_pg_log_entries": "250", 2024-09-15T03:40:56.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_min_recovery_priority": "0", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_report_interval": "5", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_num_cache_shards": "32", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_num_op_tracker_shard": "32", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_auto_affinity": "true", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_node": "-1", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_prefer_iface": "true", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objecter_finishers": "1", 2024-09-15T03:40:56.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_fuse": "false", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_tracing": "false", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_complaint_time": "30.000000", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_duration": "600", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_size": "20", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_slow_op_size": "20", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_log_threshold": "5", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards": "0", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards_hdd": "5", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards_ssd": "8", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-15T03:40:56.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_pq_min_cost": "65536", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_queue_cut_off": "high", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_thread_timeout": "15", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_open_classes_on_start": "true", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_os_flags": "0", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_peering_op_priority": "255", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_delete_cost": "1048576", 2024-09-15T03:40:56.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_delete_priority": "5", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_trim_max": "10000", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_trim_min": "100", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-15T03:40:56.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flags": "0", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_min_size": "0", 2024-09-15T03:40:56.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pg_num": "32", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pgp_num": "0", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_size": "2", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_type": "replicated", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_push_per_object_cost": "1000", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recover_clone_overlap": "true", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_cost": "20971520", 2024-09-15T03:40:56.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active": "0", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_single_start": "1", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_op_priority": "3", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_priority": "5", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep": "0.000000", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-15T03:40:56.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_repair_during_recovery": "false", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_requested_scrub_priority": "120", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_auto_repair": "false", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_begin_hour": "0", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_begin_week_day": "0", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_chunk_max": "25", 2024-09-15T03:40:56.049 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_chunk_min": "5", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_cost": "52428800", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_during_recovery": "false", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_end_hour": "0", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_end_week_day": "0", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_event_cost": "4096", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_invalid_stats": "true", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_max_preemptions": "5", 2024-09-15T03:40:56.050 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_priority": "5", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_sleep": "0.000000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_skip_data_digest": "false", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_smart_report_timeout": "5", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_cost": "1048576", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_priority": "5", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_target_transaction_size": "30", 2024-09-15T03:40:56.051 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tracing": "false", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_use_stale_snap": "false", 2024-09-15T03:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "osdc_blkin_trace_all": "false", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_kill_at": "0", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_max_join_drift": "10", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_min": "500", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_min_wait": "0.050000", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_propose_interval": "1.000000", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_max": "500", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_min": "250", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_stash_full_interval": "25", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_trim_max": "500", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_trim_min": "250", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "perf": "true", 2024-09-15T03:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "-", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrv": "", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "public_bind_addr": "-", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "public_network": "", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "public_network_interface": "", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "qat_compressor_enabled": "false", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_mon_op_timeout": "0", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_osd_op_timeout": "0", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_tracing": "false", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_atime_update_interval": "60", 2024-09-15T03:40:56.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_balance_parent_reads": "false", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_balance_snap_reads": "false", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blkin_trace_all": "false", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache": "true", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-15T03:40:56.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_policy": "writearound", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_size": "33554432", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_clone_copy_on_read": "false", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_compression_hint": "none", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_concurrent_management_ops": "10", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_clone_format": "auto", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_data_pool": "", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_features": "61", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_format": "2", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_map_options": "", 2024-09-15T03:40:56.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_order": "22", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_pool": "rbd", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_stripe_count": "0", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_stripe_unit": "0", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_enable_alloc_hint": "true", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_io_scheduler": "simple", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-15T03:40:56.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_order": "24", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_pool": "", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_splay_width": "4", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_localize_parent_reads": "false", 2024-09-15T03:40:56.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_localize_snap_reads": "false", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-15T03:40:56.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-15T03:40:56.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mtime_update_interval": "60", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_non_blocking_aio": "true", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_op_thread_timeout": "60", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_op_threads": "1", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_parent_cache_enabled": "false", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-15T03:40:56.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_plugins": "", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_burst": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_limit": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_exclude_ops": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_burst": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_limit": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-15T03:40:56.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_read_from_replica_policy": "default", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-15T03:40:56.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_skip_partial_discard": "true", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_tracing": "false", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_validate_names": "true", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_validate_pool": "true", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "restapi_base_url": "", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "restapi_log_level": "", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_acl_grants_max_num": "100", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_admin_entry": "admin", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_backend_store": "rados", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_barbican_url": "", 2024-09-15T03:40:56.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_beast_enable_async": "true", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_enabled": "true", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_expiry_interval": "900", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_lru_size": "10000", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_content_length_compat": "false", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_copy_obj_progress": "true", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cors_rules_max_num": "100", 2024-09-15T03:40:56.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cross_domain_policy": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_addr": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_password": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_username": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_require_ssl": "true", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-15T03:40:56.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-15T03:40:56.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_addr": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_auth": "token", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_namespace": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_prefix": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_token_file": "", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_buffersize": "524288", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-15T03:40:56.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_low_speed_time": "300", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-15T03:40:56.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_changes_size": "1000", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_num_shards": "128", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_window": "30", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-15T03:40:56.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-15T03:40:56.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dns_name": "", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dns_s3website_name": "", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dynamic_resharding": "true", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_gc_threads": "true", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_lc_threads": "true", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_ops_log": "false", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_quota_threads": "true", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_static_website": "false", 2024-09-15T03:40:56.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_usage_log": "false", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enforce_swift_acls": "true", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_exit_timeout_secs": "120", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_expose_bucket": "false", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_extended_http_attrs": "", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_frontends": "beast port=7480", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_deferred": "50", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_objs": "32", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-15T03:40:56.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_processor_period": "3600", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_init_timeout": "300", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_connection_idle": "300", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-15T03:40:56.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_domain": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_password": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_password_path": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_project": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_tenant": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_token": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_token_path": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_user": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_api_version": "2", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_domain": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_password": "", 2024-09-15T03:40:56.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_project": "", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_user": "", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_url": "", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_debug_interval": "-1", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_lock_max_time": "90", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_objs": "32", 2024-09-15T03:40:56.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_rules": "1000", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_worker": "3", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_thread_delay": "0", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_dnattr": "uid", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_searchfilter": "", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_uri": "", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-15T03:40:56.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_http_headers": "", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_object_name_utc": "false", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attr_name_len": "0", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attr_size": "0", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_chunk_size": "4194304", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_listing_results": "1000", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_notify_retries": "10", 2024-09-15T03:40:56.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_put_param_size": "1048576", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_put_size": "5368709120", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_slo_entries": "1000", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_md_log_max_shards": "64", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_mp_lock_max_time": "600", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-15T03:40:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_max_gc": "300", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_num_async_rados_threads": "32", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_num_control_oids": "8", 2024-09-15T03:40:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_numa_node": "-1", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_chunk_size": "100", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_gc_interval": "600", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_thread_timeout": "600", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_tracing": "false", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_token": "", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_url": "", 2024-09-15T03:40:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_verify_ssl": "true", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_file_path": "", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_rados": "true", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_socket_path": "", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_push_interval": "2.000000", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_print_continue": "true", 2024-09-15T03:40:56.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_tracing": "false", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm": "", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm_id": "", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_region": "", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-15T03:40:56.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_request_uri": "", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_batch_size": "64", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_max_aio": "128", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_num_logs": "16", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_thread_interval": "600", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_resolve_cname": "false", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_run_sync_thread": "true", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-15T03:40:56.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_scheduler_type": "throttler", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_script_uri": "", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_service_provider_name": "", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_client_id": "", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_client_secret": "", 2024-09-15T03:40:56.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_entry": "sts", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_key": "sts", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_min_session_duration": "900", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_token_introspection_url": "", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_account_in_url": "false", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_auth_entry": "auth", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_auth_url": "", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_custom_header": "", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_need_stats": "true", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_tenant_name": "", 2024-09-15T03:40:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_token_expiration": "86400", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_url": "", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_url_prefix": "swift", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_lease_period": "120", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-15T03:40:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_thread_pool_size": "512", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_comment": "", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_createby": "", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_encoding": "", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_flag": "false", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_origin": "", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_tracker": "", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_trust_forwarded_https": "false", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-15T03:40:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_max_shards": "32", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_max_user_shards": "1", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_use_opa_authz": "false", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_max_buckets": "1000", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_unique_email": "true", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_verify_ssl": "true", 2024-09-15T03:40:56.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone": "", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone_id": "", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup": "", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup_id": "", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_block_size": "4096", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-15T03:40:56.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_size": "536870912", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_index_type": "binary_search", 2024-09-15T03:40:56.091 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_partition_filters": "false", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_perf": "false", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "run_dir": "/var/run/ceph", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_block_create": "true", 2024-09-15T03:40:56.092 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_cache_lru_size": "67108864", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_default_max_object_size": "16777216", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_device_size": "107374182400", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_capacity": "16", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_segment_size": "67108864", 2024-09-15T03:40:56.093 INFO:teuthology.orchestra.run.smithi005.stdout: "setgroup": "", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "setuser": "", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "setuser_match_path": "", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "thp": "false", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "threadpool_default_timeout": "60", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout: "throttler_perf_counter": "true" 2024-09-15T03:40:56.094 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:40:56.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-15T03:40:56.217 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a: {} 2024-09-15T03:40:56.217 INFO:teuthology.orchestra.run.smithi005.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:40:56.433 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b: {} 2024-09-15T03:40:56.433 INFO:teuthology.orchestra.run.smithi005.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:40:56.447 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c: {} 2024-09-15T03:40:56.447 INFO:teuthology.orchestra.run.smithi005.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:40:56.458 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-09-15T03:40:56.477 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:40:56.477 DEBUG:teuthology.orchestra.run.smithi005:> type ceph-objectstore-tool 2024-09-15T03:40:56.492 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:40:56.493 DEBUG:teuthology.orchestra.run.smithi078:> type ceph-objectstore-tool 2024-09-15T03:40:56.511 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:40:56.511 DEBUG:teuthology.orchestra.run.smithi105:> type ceph-objectstore-tool 2024-09-15T03:40:56.528 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:40:56.529 INFO:teuthology.run_tasks:Running task rados... 2024-09-15T03:40:56.542 INFO:tasks.rados:Beginning rados... 2024-09-15T03:40:56.542 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-15T03:40:56.542 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:40:56.542 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-15T03:40:56.543 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-15T03:40:56.581 INFO:tasks.rados:joining rados 2024-09-15T03:40:56.581 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-15T03:40:56.582 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T03:40:56.582 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:40:56.582 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-15T03:40:56.583 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-15T03:40:56.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:40:56.583 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-15T03:40:56.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:40:56.584 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-15T03:40:56.584 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:40:56.584 INFO:tasks.rados:clients are ['client.0'] 2024-09-15T03:40:56.584 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-15T03:40:56.585 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:56.582+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:40:56.684 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:56.683+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:40:56.695 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:40:56.695 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:40:56.695 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:40:56.695 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:40:56.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:40:56.728 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:40:56.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:40:56.750 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:40:56.751 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:40:56.771 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:40:56.772 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:40:56.785 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:56.784+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:40:56.792 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:40:56.793 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:40:56.813 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:40:56.813 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:40:56.815 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:56.815 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:56.824 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:40:56.834 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:40:56.834 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:40:56.856 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:40:56.856 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:40:56.877 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:40:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:40:56.886 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:56.885+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:40:56.897 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:40:56.897 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:40:56.917 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:40:56.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:40:56.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:40:56.938 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:40:56.938 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:56.938 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:56.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:40:56.953 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:40:56.954 INFO:teuthology.orchestra.run.smithi005.stderr:load: shec 2024-09-15T03:40:56.960 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:40:56.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:40:56.974 INFO:tasks.rados:starting run 0 out of 1 2024-09-15T03:40:56.974 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-15T03:40:56.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-15T03:40:56.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:40:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:40:56.987 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:40:56.986+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:40:57.018 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:40:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:40:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:40:57.021+0000 7f131f7fe640 0 -- 172.21.15.5:0/1765324659 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f1320192e90 msgr2=0x7f13201952f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:40:57.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:40:57.058 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:40:57.080 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:40:57.081 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:40:57.082 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.082 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.087 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:57.087+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:40:57.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:40:57.188 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:57.187+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:40:57.235 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.236 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:40:57.289 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:57.288+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:40:57.387 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.387 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.390 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:40:57.389+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:40:57.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:40:57.492 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:57.491+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:40:57.537 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.538 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.546 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:40:57.592 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:40:57.590+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:40:57.648 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.648 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:40:57.693 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:57.692+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:40:57.794 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:40:57.793+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:40:57.799 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.799 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:40:57.895 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:40:57.894+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:40:57.909 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:57.909 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:57.918 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:40:57.996 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:57.995+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:40:58.021 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.022 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:40:58.097 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:40:58.096+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:40:58.132 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.132 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:40:58.199 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:58.198+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:40:58.254 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.255 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:40:58.300 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:40:58.299+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:40:58.367 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.367 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:40:58.421 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:58.420+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:40:58.479 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.479 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:40:58.502 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:40:58.501+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:40:58.603 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:40:58.602+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:40:58.630 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.630 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:40:58.704 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:40:58.702+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:40:58.781 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.781 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:40:58.838 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:40:58.837+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:40:58.894 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:58.895 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:58.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:40:58.905 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:40:58.905+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:40:58.963 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_0' created 2024-09-15T03:40:58.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:40:58.984 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-15T03:40:59.006 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:40:59.005+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:40:59.053 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:59.054 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:59.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:40:59.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:40:59.107 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:59.106+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:40:59.168 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:59.169 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:59.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:40:59.208 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:59.207+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:40:59.227 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:40:59.227 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:40:59.248 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:40:59.248 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:40:59.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:40:59.269 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:40:59.278 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:59.279 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:59.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:40:59.289 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:40:59.289 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:40:59.308 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:59.307+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:40:59.309 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:40:59.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:40:59.329 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:40:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:40:59.350 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:40:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:40:59.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:40:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:40:59.389 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:40:59.390 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:40:59.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:40:59.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:40:59.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:40:59.409 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:40:59.408+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:40:59.410 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:40:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:40:59.431 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:40:59.431 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:40:59.452 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:40:59.452 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:40:59.472 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:40:59.472 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:40:59.492 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:40:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:40:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:40:59.496+0000 7f80777fe640 0 -- 172.21.15.5:0/660298844 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f8078195580 msgr2=0x7f8078197960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:40:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:40:59.500 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:40:59.500 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:40:59.500 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:40:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:40:59.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:40:59.510 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:40:59.509+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:40:59.610 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:40:59.610+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:40:59.613 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.665198+0000", 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.946053114, 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010431241000000001, 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 25 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.665198+0000", 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.665198+0000", 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.665199+0000", 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8199999999999999e-07 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.665204+0000", 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1139999999999997e-06 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.665211+0000", 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.61e-06 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.665216+0000", 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5240000000000002e-06 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.670209+0000", 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049931209999999997 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.670289+0000", 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0062999999999996e-05 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.670409+0000", 2024-09-15T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119889 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.671190+0000", 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078160000000000002 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.675566+0000", 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043759599999999999 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.675606+0000", 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9425000000000003e-05 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.675629+0000", 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3153e-05 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849251+0000", 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761999937000001, 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089718030000000004, 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849251+0000", 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849251+0000", 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849251+0000", 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849255+0000", 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7979999999999999e-06 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849261+0000", 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1009999999999997e-06 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849264+0000", 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8509999999999999e-06 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.852310+0000", 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030463740000000001 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.852362+0000", 2024-09-15T03:40:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1347999999999997e-05 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.852449+0000", 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7427000000000002e-05 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.853757+0000", 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001308018 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858162+0000", 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044043809999999997 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858200+0000", 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8776000000000003e-05 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858223+0000", 2024-09-15T03:40:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2368e-05 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849267+0000", 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761983709000001, 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090089290000000006, 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 47 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849267+0000", 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849267+0000", 2024-09-15T03:40:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849267+0000", 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.29e-07 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849317+0000", 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9249999999999998e-05 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849324+0000", 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8870000000000003e-06 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849327+0000", 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-15T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.852776+0000", 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034486249999999999 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.852828+0000", 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1956000000000002e-05 2024-09-15T03:40:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.852925+0000", 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7503999999999993e-05 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.857374+0000", 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044488840000000002 2024-09-15T03:40:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858230+0000", 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000856159 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858255+0000", 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5236999999999999e-05 2024-09-15T03:40:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858276+0000", 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0692000000000001e-05 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849332+0000", 2024-09-15T03:40:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761918969, 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089916400000000004, 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 48 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849332+0000", 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849332+0000", 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849332+0000", 2024-09-15T03:40:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6300000000000001e-07 2024-09-15T03:40:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849335+0000", 2024-09-15T03:40:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3649999999999998e-06 2024-09-15T03:40:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849340+0000", 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0100000000000003e-06 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849342+0000", 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.413e-06 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.853787+0000", 2024-09-15T03:40:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044450310000000003 2024-09-15T03:40:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.853841+0000", 2024-09-15T03:40:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3566000000000002e-05 2024-09-15T03:40:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.853937+0000", 2024-09-15T03:40:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6430999999999999e-05 2024-09-15T03:40:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.857407+0000", 2024-09-15T03:40:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034700629999999998 2024-09-15T03:40:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858281+0000", 2024-09-15T03:40:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087403700000000003 2024-09-15T03:40:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858305+0000", 2024-09-15T03:40:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3325000000000001e-05 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858324+0000", 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9035999999999999e-05 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849345+0000", 2024-09-15T03:40:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761906226000001, 2024-09-15T03:40:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010669869, 2024-09-15T03:40:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 49 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849345+0000", 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849345+0000", 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849345+0000", 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7599999999999998e-07 2024-09-15T03:40:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849347+0000", 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.902e-06 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849352+0000", 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5350000000000004e-06 2024-09-15T03:40:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849357+0000", 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.78e-06 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854237+0000", 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048793639999999998 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854319+0000", 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2861000000000005e-05 2024-09-15T03:40:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854457+0000", 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137422 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.858330+0000", 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038732020000000001 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.859896+0000", 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015663459999999999 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.859949+0000", 2024-09-15T03:40:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2803999999999998e-05 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860015+0000", 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5376999999999994e-05 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761888661, 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013249957, 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 50 2024-09-15T03:40:59.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849363+0000", 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2099999999999997e-07 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849419+0000", 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6660999999999998e-05 2024-09-15T03:40:59.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849432+0000", 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2313999999999999e-05 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849437+0000", 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6130000000000003e-06 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854814+0000", 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053769459999999996 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854864+0000", 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9937e-05 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854960+0000", 2024-09-15T03:40:59.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6176999999999998e-05 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860859+0000", 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058984600000000003 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862557+0000", 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016980459999999999 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862590+0000", 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3015999999999997e-05 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862612+0000", 2024-09-15T03:40:59.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2266e-05 2024-09-15T03:40:59.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.664 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.664 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:40:59.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761797268, 2024-09-15T03:40:59.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:40:59.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.665 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:40:59.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:40:59.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:40:59.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:40:59.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:40:59.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:40:59.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:40:59.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:40:59.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:40:59.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:40:59.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:40:59.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:40:59.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:40:59.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:40:59.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:40:59.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:40:59.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:40:59.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:40:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:40:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:40:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:40:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:40:59.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:40:59.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:40:59.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:40:59.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.679 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.679 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:40:59.679 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.761769163, 2024-09-15T03:40:59.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:40:59.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.680 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.681 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:40:59.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.681 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:40:59.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:40:59.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:40:59.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:40:59.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:40:59.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:40:59.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:40:59.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:40:59.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:40:59.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:40:59.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:40:59.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:40:59.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:40:59.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:40:59.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:40:59.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:40:59.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:40:59.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:40:59.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:40:59.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:40:59.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:40:59.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:40:59.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:40:59.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:40:59.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.693 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.694 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:55 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.694 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:40:59.694 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.747973051000001, 2024-09-15T03:40:59.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011236095999999999, 2024-09-15T03:40:59.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.695 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.695 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.696 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 55 2024-09-15T03:40:59.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.696 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:40:59.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:40:59.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:40:59.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:40:59.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863353+0000", 2024-09-15T03:40:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4338999999999994e-05 2024-09-15T03:40:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863359+0000", 2024-09-15T03:40:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000003e-06 2024-09-15T03:40:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863364+0000", 2024-09-15T03:40:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3649999999999997e-06 2024-09-15T03:40:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867028+0000", 2024-09-15T03:40:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036646890000000001 2024-09-15T03:40:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867081+0000", 2024-09-15T03:40:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2984e-05 2024-09-15T03:40:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867157+0000", 2024-09-15T03:40:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6079000000000001e-05 2024-09-15T03:40:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.870092+0000", 2024-09-15T03:40:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029341810000000001 2024-09-15T03:40:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.874426+0000", 2024-09-15T03:40:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043345409999999999 2024-09-15T03:40:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.874474+0000", 2024-09-15T03:40:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8285000000000003e-05 2024-09-15T03:40:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.874514+0000", 2024-09-15T03:40:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9508999999999997e-05 2024-09-15T03:40:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:56 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:40:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.747882382, 2024-09-15T03:40:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012488551000000001, 2024-09-15T03:40:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 56 2024-09-15T03:40:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:40:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:40:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863369+0000", 2024-09-15T03:40:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:40:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863435+0000", 2024-09-15T03:40:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6039999999999998e-05 2024-09-15T03:40:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863441+0000", 2024-09-15T03:40:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8250000000000001e-06 2024-09-15T03:40:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863444+0000", 2024-09-15T03:40:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2739999999999999e-06 2024-09-15T03:40:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867710+0000", 2024-09-15T03:40:59.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042658740000000002 2024-09-15T03:40:59.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867818+0000", 2024-09-15T03:40:59.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010801100000000001 2024-09-15T03:40:59.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867963+0000", 2024-09-15T03:40:59.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014490799999999999 2024-09-15T03:40:59.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.872812+0000", 2024-09-15T03:40:59.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048489290000000001 2024-09-15T03:40:59.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875774+0000", 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029621589999999998 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875821+0000", 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6966999999999999e-05 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875857+0000", 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6059999999999997e-05 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.012302+0000", 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.59894879, 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008805851, 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.725 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012302+0000", 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012302+0000", 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012302+0000", 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7e-07 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012383+0000", 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599000000000005e-05 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012391+0000", 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0469999999999994e-06 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012396+0000", 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4800000000000003e-06 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.016258+0000", 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003862351 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.016298+0000", 2024-09-15T03:40:59.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0058999999999997e-05 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.016372+0000", 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4294000000000004e-05 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.018838+0000", 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024653700000000001 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.021069+0000", 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022310839999999999 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.021090+0000", 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1753e-05 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.021108+0000", 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7444000000000001e-05 2024-09-15T03:40:59.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.012401+0000", 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.598849383999999, 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087656350000000008, 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012401+0000", 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012401+0000", 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012402+0000", 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012406+0000", 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012413+0000", 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3890000000000002e-06 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.012418+0000", 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.339e-06 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.018182+0000", 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057636249999999997 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.018356+0000", 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174248 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.018476+0000", 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012024 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.019063+0000", 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058685499999999995 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.021131+0000", 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020685730000000002 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.021151+0000", 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9783e-05 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.021167+0000", 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5911000000000001e-05 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.738 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.143962+0000", 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.467289141, 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010792263999999999, 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.143962+0000", 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.143962+0000", 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.143962+0000", 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7199999999999999e-07 2024-09-15T03:40:59.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144068+0000", 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010616900000000001 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144078+0000", 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7759999999999992e-06 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144083+0000", 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0150000000000003e-06 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.148430+0000", 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00434673 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.148473+0000", 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2719000000000002e-05 2024-09-15T03:40:59.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.148543+0000", 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0192999999999997e-05 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152111+0000", 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035676470000000002 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154696+0000", 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025850809999999999 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154723+0000", 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7132999999999999e-05 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154754+0000", 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1229000000000002e-05 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.744 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.745 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.745 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144099+0000", 2024-09-15T03:40:59.745 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.46715165, 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010719414, 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144099+0000", 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144099+0000", 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144100+0000", 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4799999999999998e-07 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144104+0000", 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4490000000000001e-06 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144112+0000", 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1130000000000001e-06 2024-09-15T03:40:59.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144117+0000", 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2740000000000001e-06 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.148847+0000", 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047298790000000002 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.148898+0000", 2024-09-15T03:40:59.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1517999999999997e-05 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.148970+0000", 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1587000000000004e-05 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152271+0000", 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033009100000000002 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154762+0000", 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024918739999999998 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154786+0000", 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-05 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154819+0000", 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3092000000000001e-05 2024-09-15T03:40:59.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144122+0000", 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.467128657, 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010764991, 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 95 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144122+0000", 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144122+0000", 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144123+0000", 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.75e-07 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144128+0000", 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6720000000000001e-06 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144136+0000", 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0479999999999999e-06 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144140+0000", 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7510000000000002e-06 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.149195+0000", 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050552150000000001 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.149238+0000", 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3071999999999997e-05 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.149309+0000", 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0203000000000005e-05 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152300+0000", 2024-09-15T03:40:59.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002990936 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154824+0000", 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025248240000000002 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154846+0000", 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1676e-05 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154887+0000", 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1118999999999997e-05 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144145+0000", 2024-09-15T03:40:59.758 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.467105381, 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010829622000000001, 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 96 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144145+0000", 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144145+0000", 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144146+0000", 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9800000000000004e-07 2024-09-15T03:40:59.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144212+0000", 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5699999999999998e-05 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144224+0000", 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2461e-05 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144229+0000", 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7400000000000004e-06 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.149523+0000", 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052943929999999997 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.149568+0000", 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4297e-05 2024-09-15T03:40:59.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.149641+0000", 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3114999999999994e-05 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154893+0000", 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005252151 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154928+0000", 2024-09-15T03:40:59.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4786000000000002e-05 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154955+0000", 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7532e-05 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154975+0000", 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9949000000000002e-05 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.467001049, 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012203834, 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 97 2024-09-15T03:40:59.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1699999999999998e-07 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144254+0000", 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8039999999999999e-06 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144259+0000", 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3310000000000003e-06 2024-09-15T03:40:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144262+0000", 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490000000000002e-06 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151586+0000", 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0073231959999999997 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151660+0000", 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4589e-05 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151733+0000", 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2595000000000005e-05 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154896+0000", 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031635869999999998 2024-09-15T03:40:59.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156408+0000", 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001511996 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156435+0000", 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6917999999999999e-05 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156454+0000", 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8451999999999999e-05 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.466985727999999, 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012289876999999999, 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 98 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.34e-07 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144267+0000", 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.795e-06 2024-09-15T03:40:59.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144270+0000", 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1760000000000002e-06 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144273+0000", 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152322+0000", 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080492889999999994 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152365+0000", 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3634999999999998e-05 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152434+0000", 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8738999999999994e-05 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154902+0000", 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024674879999999999 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156492+0000", 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015906309999999999 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156520+0000", 2024-09-15T03:40:59.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.811e-05 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156555+0000", 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4585000000000002e-05 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:40:59.777 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.466975915000001, 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013168571, 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 99 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:40:59.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144313+0000", 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7694000000000002e-05 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144317+0000", 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2719999999999999e-06 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144320+0000", 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152651+0000", 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083311770000000004 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152696+0000", 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4750999999999999e-05 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152767+0000", 2024-09-15T03:40:59.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0850000000000001e-05 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156577+0000", 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038109319999999999 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157385+0000", 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080757500000000005 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157410+0000", 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5306000000000001e-05 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157444+0000", 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3192999999999999e-05 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.783 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.466928057000001, 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:40:59.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:40:59.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:40:59.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:40:59.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:40:59.790 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:40:59.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:40:59.792 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:40:59.790+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:40:59.892 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:40:59.891+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:40:59.900 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:40:59.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:40:59.901 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:40:59.901 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:40:59.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:40:59.993 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:40:59.992+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:40:59.995 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-15T03:40:59.996 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:41:00.009 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:00.009 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:00.094 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:00.093+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:00.116 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:47 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'29, mlcod=24'20)", 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:43.853009+0000", 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.261492562000001, 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041529840000000002, 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853009+0000", 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853009+0000", 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853010+0000", 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853079+0000", 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9770000000000005e-05 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853097+0000", 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7856e-05 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853100+0000", 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1559999999999999e-06 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853129+0000", 2024-09-15T03:41:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8226e-05 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.853147+0000", 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8142e-05 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.857138+0000", 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039915369999999999 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.857162+0000", 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3864000000000001e-05 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:49 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'31, mlcod=24'23)", 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:43.854550+0000", 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.259951328, 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050498899999999996, 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.854550+0000", 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.854550+0000", 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.854551+0000", 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1699999999999997e-07 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.854556+0000", 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6979999999999997e-06 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.854583+0000", 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7385e-05 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.854588+0000", 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4379999999999997e-06 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856361+0000", 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017736640000000001 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856375+0000", 2024-09-15T03:41:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3672000000000001e-05 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.859571+0000", 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003196445 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.859600+0000", 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8870999999999999e-05 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:50 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'32, mlcod=24'23)", 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:43.855076+0000", 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.259425275000002, 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073421220000000004, 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855076+0000", 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855076+0000", 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855077+0000", 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855126+0000", 2024-09-15T03:41:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9027999999999999e-05 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855154+0000", 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8011999999999999e-05 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855159+0000", 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3590000000000001e-06 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856555+0000", 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001396226 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856574+0000", 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9476000000000001e-05 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862388+0000", 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005813688 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862419+0000", 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0604e-05 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:51 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'33, mlcod=24'23)", 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:43.855779+0000", 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.258723094, 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066606879999999997, 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855779+0000", 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855779+0000", 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855779+0000", 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855785+0000", 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.925e-06 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855813+0000", 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7294999999999999e-05 2024-09-15T03:41:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855817+0000", 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.262e-06 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856994+0000", 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011775609999999999 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.857006+0000", 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2003e-05 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862422+0000", 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005415375 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862439+0000", 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7521999999999999e-05 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:52 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'34, mlcod=24'23)", 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:43.856326+0000", 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.258176004999999, 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006127207, 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856326+0000", 2024-09-15T03:41:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856326+0000", 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856326+0000", 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856331+0000", 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.899e-06 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856364+0000", 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3188000000000001e-05 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856369+0000", 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5240000000000002e-06 2024-09-15T03:41:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.857193+0000", 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000824036 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.857206+0000", 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3151e-05 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862440+0000", 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052337870000000002 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862453+0000", 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2892e-05 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:66 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'44, mlcod=24'40)", 2024-09-15T03:41:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.101961455000001, 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077416819999999997, 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012541+0000", 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2e-07 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012592+0000", 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1273999999999997e-05 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012612+0000", 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0389000000000001e-05 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012615+0000", 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1250000000000001e-06 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012636+0000", 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1070999999999999e-05 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012647+0000", 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.08e-05 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020245+0000", 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075975579999999999 2024-09-15T03:41:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020282+0000", 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7045000000000002e-05 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:67 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'45, mlcod=24'40)", 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.101634954000001, 2024-09-15T03:41:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074323139999999998, 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012870+0000", 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012888+0000", 2024-09-15T03:41:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8046e-05 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012891+0000", 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0170000000000001e-06 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012916+0000", 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4307e-05 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012928+0000", 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1956e-05 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020284+0000", 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073567820000000001 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020299+0000", 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4555e-05 2024-09-15T03:41:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:68 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'46, mlcod=24'40)", 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.101229074999999, 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070388200000000003, 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013276+0000", 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013293+0000", 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6504999999999999e-05 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013295+0000", 2024-09-15T03:41:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7599999999999998e-06 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013319+0000", 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3272999999999999e-05 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013330+0000", 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020300+0000", 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069692130000000001 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020311+0000", 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.173e-05 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:69 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'47, mlcod=24'40)", 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.013574+0000", 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.100928072999999, 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067494349999999998, 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013574+0000", 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013574+0000", 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013574+0000", 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4099999999999999e-07 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013623+0000", 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9120999999999999e-05 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013642+0000", 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8791e-05 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013646+0000", 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013668+0000", 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2042999999999999e-05 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013679+0000", 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0875e-05 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020312+0000", 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066328250000000002 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020323+0000", 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0949e-05 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:70 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'48, mlcod=24'40)", 2024-09-15T03:41:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.013897+0000", 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.100604836999999, 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064365739999999996, 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013897+0000", 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013897+0000", 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013897+0000", 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2300000000000002e-07 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013902+0000", 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4460000000000003e-06 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013919+0000", 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7407000000000001e-05 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013922+0000", 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9340000000000002e-06 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013945+0000", 2024-09-15T03:41:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3218999999999999e-05 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013957+0000", 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1418e-05 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020324+0000", 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063670619999999997 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020333+0000", 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6649999999999998e-06 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:71 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'49, mlcod=24'40)", 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.014344+0000", 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.100157744000001, 2024-09-15T03:41:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005998247, 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014344+0000", 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014344+0000", 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014344+0000", 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6400000000000002e-07 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014349+0000", 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6970000000000001e-06 2024-09-15T03:41:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014370+0000", 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0356999999999998e-05 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014373+0000", 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3900000000000002e-06 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014397+0000", 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4134e-05 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014408+0000", 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1245e-05 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020334+0000", 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059257090000000004 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020342+0000", 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1510000000000004e-06 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:72 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'50, mlcod=24'40)", 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.014812+0000", 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.09968946, 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005827456, 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014812+0000", 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014812+0000", 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014813+0000", 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9999999999999998e-07 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014874+0000", 2024-09-15T03:41:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1190000000000002e-05 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014894+0000", 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0105000000000001e-05 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014898+0000", 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2440000000000004e-06 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014921+0000", 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2889999999999999e-05 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.014936+0000", 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5167999999999999e-05 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020606+0000", 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005669449 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020640+0000", 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3909999999999999e-05 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:73 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'51, mlcod=24'40)", 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.015496+0000", 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.099005239, 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051640990000000001, 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015496+0000", 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015496+0000", 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015497+0000", 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015501+0000", 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7890000000000001e-06 2024-09-15T03:41:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015521+0000", 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0665999999999998e-05 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015525+0000", 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6909999999999999e-06 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015549+0000", 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4192000000000001e-05 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015562+0000", 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3132000000000001e-05 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020642+0000", 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050791849999999999 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020661+0000", 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8952000000000001e-05 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:74 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'52, mlcod=24'40)", 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.015956+0000", 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.098546118000002, 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047239889999999996, 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015956+0000", 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015956+0000", 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015956+0000", 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0900000000000002e-07 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015960+0000", 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4159999999999997e-06 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015978+0000", 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7059999999999999e-05 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.015980+0000", 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.779e-06 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016001+0000", 2024-09-15T03:41:00.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0554e-05 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016013+0000", 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2123000000000001e-05 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020662+0000", 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046491859999999996 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020680+0000", 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7362000000000001e-05 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'41)", 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.016472+0000", 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.098029351000001, 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042241980000000002, 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016472+0000", 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016472+0000", 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016473+0000", 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1500000000000005e-07 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016524+0000", 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0991e-05 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016544+0000", 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0007999999999999e-05 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016547+0000", 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5750000000000001e-06 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016570+0000", 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2867e-05 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.016582+0000", 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1252e-05 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020680+0000", 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040988650000000001 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020697+0000", 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6124999999999999e-05 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:90 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'64, mlcod=24'60)", 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.144825+0000", 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.969676416, 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055055340000000003, 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144825+0000", 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144825+0000", 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144826+0000", 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144895+0000", 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9590000000000003e-05 2024-09-15T03:41:00.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144933+0000", 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7248e-05 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144938+0000", 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4820000000000002e-06 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144961+0000", 2024-09-15T03:41:00.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2538e-05 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.144971+0000", 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.044e-05 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.150275+0000", 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053037140000000002 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.150331+0000", 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6002999999999999e-05 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:96 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'70, mlcod=24'62)", 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.149713+0000", 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.964788854, 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038057020000000002, 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149713+0000", 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149713+0000", 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149713+0000", 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149770+0000", 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6421000000000001e-05 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149784+0000", 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4377e-05 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149787+0000", 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.706e-06 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149809+0000", 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2143000000000001e-05 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.149819+0000", 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0444e-05 2024-09-15T03:41:00.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.153495+0000", 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036754769999999999 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.153519+0000", 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3849000000000002e-05 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:97 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'71, mlcod=24'62)", 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.151791+0000", 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.962711136999999, 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044605820000000003, 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151791+0000", 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151791+0000", 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151791+0000", 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151803+0000", 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2116999999999999e-05 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151821+0000", 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7623999999999999e-05 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151824+0000", 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7409999999999998e-06 2024-09-15T03:41:00.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151845+0000", 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0886999999999998e-05 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.151857+0000", 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1647999999999999e-05 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.156227+0000", 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043699209999999997 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.156251+0000", 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4153999999999999e-05 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:98 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'65)", 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.152558+0000", 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.961943740000001, 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037065800000000001, 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152558+0000", 2024-09-15T03:41:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152558+0000", 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152559+0000", 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:41:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152570+0000", 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0453e-05 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152632+0000", 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2470000000000003e-05 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152643+0000", 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0243e-05 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152720+0000", 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7322000000000003e-05 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.152752+0000", 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2396e-05 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.156253+0000", 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035007409999999999 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.156264+0000", 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1415e-05 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.206 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:100 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'74, mlcod=24'65)", 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.154603+0000", 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.95989835, 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072170230000000004, 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154603+0000", 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154603+0000", 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154604+0000", 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6700000000000003e-07 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154625+0000", 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1237e-05 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154648+0000", 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2807999999999999e-05 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154652+0000", 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.07e-06 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154675+0000", 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3394e-05 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154687+0000", 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1858999999999999e-05 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.161792+0000", 2024-09-15T03:41:00.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071049260000000001 2024-09-15T03:41:00.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.161820+0000", 2024-09-15T03:41:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8161999999999999e-05 2024-09-15T03:41:00.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:00.213 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:00.213 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.214 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:00.215 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:00.213+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:00.315 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:00.314+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:00.356 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.356 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:00.356 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:00.356 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:00.417 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:00.416+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:00.433 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:00.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:00.454 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:00.454 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:00.464 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:00.465 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:00.518 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:00.517+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:00.616 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.616 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:00.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:00.616 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [] 2024-09-15T03:41:00.617 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.621 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:00.618+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:00.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:00.720 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:00.719+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:00.724 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:00.724 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:00.820 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:00.819+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:00.833 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:00.833 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:00.833 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:00.922 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:00.921+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:00.940 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:00.940 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:00.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:00.940 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [] 2024-09-15T03:41:00.940 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:00.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:01.016 DEBUG:teuthology.orchestra.run.smithi105:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-15T03:41:01.022 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:01.021+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:01.045 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight read -> 100 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight delete -> 50 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight snap_create -> 50 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight snap_remove -> 50 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight rollback -> 50 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight setattr -> 25 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight rmattr -> 25 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight copy_from -> 50 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight append -> 100 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable) 2024-09-15T03:41:01.046 INFO:tasks.rados.rados.0.smithi105.stdout:Configuration: 2024-09-15T03:41:01.047 INFO:tasks.rados.rados.0.smithi105.stdout: Number of operations: 400 2024-09-15T03:41:01.047 INFO:tasks.rados.rados.0.smithi105.stdout: Number of objects: 50 2024-09-15T03:41:01.047 INFO:tasks.rados.rados.0.smithi105.stdout: Max in flight operations: 16 2024-09-15T03:41:01.047 INFO:tasks.rados.rados.0.smithi105.stdout: Object size (in bytes): 4000000 2024-09-15T03:41:01.047 INFO:tasks.rados.rados.0.smithi105.stdout: Write stride min: 400000 2024-09-15T03:41:01.047 INFO:tasks.rados.rados.0.smithi105.stdout: Write stride max: 800000 2024-09-15T03:41:01.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:01.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:01.048 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:01.048 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:01.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:01.141 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:01.140+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:01.196 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:01.196 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:01.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:01.223 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:01.223+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:01.325 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:01.324+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:01.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:01.345 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:01.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:01.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [] 2024-09-15T03:41:01.345 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:01.354 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:01.425 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:01.424+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:01.463 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:01.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:01.463 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:01.463 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:01.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:41:01.526 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:01.525+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:01.569 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:01.569 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:01.569 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:01.569 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:01.569 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:01.577 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:41:01.582 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T03:41:01.583 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:41:01.583 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-15T03:41:01.583 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T03:41:01.583 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T03:41:01.583 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-15T03:41:01.627 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:01.626+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:01.674 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4308.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:40:46.672017+0000", 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 15.001946956999999, 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.103760043, 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:01.675 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4308", 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.5:0/4091361720", 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672017+0000", 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672017+0000", 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672030+0000", 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.278e-05 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672041+0000", 2024-09-15T03:41:01.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.112e-05 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672064+0000", 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3459e-05 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672081+0000", 2024-09-15T03:41:01.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6405999999999998e-05 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672185+0000", 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104077 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:01.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.672192+0000", 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4100000000000002e-06 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.775617+0000", 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.103424513 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.775777+0000", 2024-09-15T03:41:01.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000160278 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4308.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:40:46.776058+0000", 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.897905899, 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000142657, 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:01.681 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4308", 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.5:0/4091361720", 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776058+0000", 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776058+0000", 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776059+0000", 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8100000000000001e-07 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776064+0000", 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2669999999999997e-06 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776078+0000", 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.412e-05 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776086+0000", 2024-09-15T03:41:01.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8169999999999999e-06 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776120+0000", 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4208999999999997e-05 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:46.776201+0000", 2024-09-15T03:41:01.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0263e-05 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4308.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:40:47.657347+0000", 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.016616523, 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0036322220000000001, 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:01.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:01.687 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4308", 2024-09-15T03:41:01.687 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.5:0/4091361720", 2024-09-15T03:41:01.687 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 5 2024-09-15T03:41:01.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.687 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:01.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:01.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657347+0000", 2024-09-15T03:41:01.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657347+0000", 2024-09-15T03:41:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657349+0000", 2024-09-15T03:41:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.367e-06 2024-09-15T03:41:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657356+0000", 2024-09-15T03:41:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-09-15T03:41:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657369+0000", 2024-09-15T03:41:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3131e-05 2024-09-15T03:41:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657377+0000", 2024-09-15T03:41:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6109999999999996e-06 2024-09-15T03:41:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657438+0000", 2024-09-15T03:41:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1100999999999997e-05 2024-09-15T03:41:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op must wait for map", 2024-09-15T03:41:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.657443+0000", 2024-09-15T03:41:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0579999999999997e-06 2024-09-15T03:41:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.659176+0000", 2024-09-15T03:41:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001732786 2024-09-15T03:41:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.659274+0000", 2024-09-15T03:41:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8367000000000002e-05 2024-09-15T03:41:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 12", 2024-09-15T03:41:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.659394+0000", 2024-09-15T03:41:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119731 2024-09-15T03:41:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-09-15T03:41:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660154+0000", 2024-09-15T03:41:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076014300000000002 2024-09-15T03:41:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660918+0000", 2024-09-15T03:41:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076356600000000003 2024-09-15T03:41:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660954+0000", 2024-09-15T03:41:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6091999999999999e-05 2024-09-15T03:41:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660980+0000", 2024-09-15T03:41:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5687999999999999e-05 2024-09-15T03:41:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4308.0:5 2.2 e27/26 ondisk, result = 0)", 2024-09-15T03:41:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:40:47.660825+0000", 2024-09-15T03:41:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.01313843, 2024-09-15T03:41:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016023799999999999, 2024-09-15T03:41:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660825+0000", 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660825+0000", 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660827+0000", 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0690000000000001e-06 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660832+0000", 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3329999999999996e-06 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660864+0000", 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2372000000000001e-05 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660870+0000", 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.976e-06 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660907+0000", 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6399000000000002e-05 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660914+0000", 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9399999999999996e-06 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:47.660986+0000", 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2149000000000003e-05 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4308.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:40:48.664689+0000", 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.009274966, 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003837086, 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4308", 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.5:0/4091361720", 2024-09-15T03:41:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 7 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664689+0000", 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664689+0000", 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664691+0000", 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.823e-06 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664699+0000", 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2609999999999993e-06 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664714+0000", 2024-09-15T03:41:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5021000000000001e-05 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664721+0000", 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0859999999999996e-06 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664809+0000", 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7868999999999995e-05 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op must wait for map", 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.664817+0000", 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3950000000000001e-06 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.666595+0000", 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017771880000000001 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.666689+0000", 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4141999999999994e-05 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 12", 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.666802+0000", 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113307 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668029+0000", 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012274790000000001 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668467+0000", 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043751899999999998 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668499+0000", 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2369999999999997e-05 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668526+0000", 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6625999999999999e-05 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4308.0:7 2.2 e28/26 ondisk, result = 0)", 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:40:48.668392+0000", 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.005571661999999, 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000138911, 2024-09-15T03:41:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668392+0000", 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668392+0000", 2024-09-15T03:41:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668393+0000", 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1289999999999999e-06 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668398+0000", 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6840000000000004e-06 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668426+0000", 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7730999999999999e-05 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668431+0000", 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6529999999999999e-06 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668457+0000", 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5865000000000001e-05 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668462+0000", 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1420000000000001e-06 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:40:48.668531+0000", 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8707000000000005e-05 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:01.722 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:01.724 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:01.724 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T03:41:01.724 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:01.724 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:01.728 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:01.726+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:01.822 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:01.823 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:01.828 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:01.827+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:01.831 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:01.929 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:01.928+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:01.968 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:01.968 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:01.968 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:01.969 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:01.969 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:01.976 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:02.012 INFO:tasks.rados.rados.0.smithi105.stdout:1: write initial oid 1 2024-09-15T03:41:02.012 INFO:tasks.rados.rados.0.smithi105.stdout:1: seq_num 0 ranges {0=32768} 2024-09-15T03:41:02.012 INFO:tasks.rados.rados.0.smithi105.stdout:1: writing smithi10536496-1 from 0 to 32768 tid 1 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:2: write initial oid 2 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:2: seq_num 1 ranges {0=32768} 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:2: writing smithi10536496-2 from 0 to 32768 tid 1 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:3: write initial oid 3 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:3: seq_num 2 ranges {0=49152} 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:3: writing smithi10536496-3 from 0 to 49152 tid 1 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:4: write initial oid 4 2024-09-15T03:41:02.013 INFO:tasks.rados.rados.0.smithi105.stdout:4: seq_num 3 ranges {0=65536} 2024-09-15T03:41:02.015 INFO:tasks.rados.rados.0.smithi105.stdout:4: writing smithi10536496-4 from 0 to 65536 tid 1 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:5: write initial oid 5 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:5: seq_num 4 ranges {0=81920} 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:5: writing smithi10536496-5 from 0 to 81920 tid 1 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:6: write initial oid 6 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:6: seq_num 5 ranges {0=98304} 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:6: writing smithi10536496-6 from 0 to 98304 tid 1 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:7: write initial oid 7 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:7: seq_num 6 ranges {0=114688} 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:7: writing smithi10536496-7 from 0 to 114688 tid 1 2024-09-15T03:41:02.016 INFO:tasks.rados.rados.0.smithi105.stdout:8: write initial oid 8 2024-09-15T03:41:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:8: seq_num 7 ranges {0=131072} 2024-09-15T03:41:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:8: writing smithi10536496-8 from 0 to 131072 tid 1 2024-09-15T03:41:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:9: write initial oid 9 2024-09-15T03:41:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:9: seq_num 8 ranges {0=147456} 2024-09-15T03:41:02.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:41:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:9: writing smithi10536496-9 from 0 to 147456 tid 1 2024-09-15T03:41:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:10: write initial oid 10 2024-09-15T03:41:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:10: seq_num 9 ranges {0=163840} 2024-09-15T03:41:02.019 INFO:tasks.rados.rados.0.smithi105.stdout:10: writing smithi10536496-10 from 0 to 163840 tid 1 2024-09-15T03:41:02.019 INFO:tasks.rados.rados.0.smithi105.stdout:11: write initial oid 11 2024-09-15T03:41:02.019 INFO:tasks.rados.rados.0.smithi105.stdout:11: seq_num 10 ranges {0=180224} 2024-09-15T03:41:02.019 INFO:tasks.rados.rados.0.smithi105.stdout:11: writing smithi10536496-11 from 0 to 180224 tid 1 2024-09-15T03:41:02.019 INFO:tasks.rados.rados.0.smithi105.stdout:12: write initial oid 12 2024-09-15T03:41:02.019 INFO:tasks.rados.rados.0.smithi105.stdout:12: seq_num 11 ranges {0=196608} 2024-09-15T03:41:02.021 INFO:tasks.rados.rados.0.smithi105.stdout:12: writing smithi10536496-12 from 0 to 196608 tid 1 2024-09-15T03:41:02.021 INFO:tasks.rados.rados.0.smithi105.stdout:13: write initial oid 13 2024-09-15T03:41:02.021 INFO:tasks.rados.rados.0.smithi105.stdout:13: seq_num 12 ranges {0=212992} 2024-09-15T03:41:02.023 INFO:tasks.rados.rados.0.smithi105.stdout:13: writing smithi10536496-13 from 0 to 212992 tid 1 2024-09-15T03:41:02.023 INFO:tasks.rados.rados.0.smithi105.stdout:14: write initial oid 14 2024-09-15T03:41:02.024 INFO:tasks.rados.rados.0.smithi105.stdout:14: seq_num 13 ranges {0=229376} 2024-09-15T03:41:02.025 INFO:tasks.rados.rados.0.smithi105.stdout:14: writing smithi10536496-14 from 0 to 229376 tid 1 2024-09-15T03:41:02.025 INFO:tasks.rados.rados.0.smithi105.stdout:15: write initial oid 15 2024-09-15T03:41:02.026 INFO:tasks.rados.rados.0.smithi105.stdout:15: seq_num 14 ranges {0=245760} 2024-09-15T03:41:02.027 INFO:tasks.rados.rados.0.smithi105.stdout:15: writing smithi10536496-15 from 0 to 245760 tid 1 2024-09-15T03:41:02.028 INFO:tasks.rados.rados.0.smithi105.stdout:16: write initial oid 16 2024-09-15T03:41:02.028 INFO:tasks.rados.rados.0.smithi105.stdout:16: seq_num 15 ranges {0=262144} 2024-09-15T03:41:02.030 INFO:tasks.rados.rados.0.smithi105.stdout:16: writing smithi10536496-16 from 0 to 262144 tid 1 2024-09-15T03:41:02.030 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.030 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 1 to smithi10536496-2 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 1 to smithi10536496-4 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 2 to smithi10536496-4 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 2 to smithi10536496-2 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 1 to smithi10536496-1 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 2 to smithi10536496-1 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 3 to smithi10536496-2 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 3 to smithi10536496-4 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 1 to smithi10536496-3 2024-09-15T03:41:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 2 to smithi10536496-3 2024-09-15T03:41:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:2: done (15 left) 2024-09-15T03:41:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:4: done (14 left) 2024-09-15T03:41:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:17: write initial oid 17 2024-09-15T03:41:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:17: seq_num 16 ranges {0=278528} 2024-09-15T03:41:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:17: writing smithi10536496-17 from 0 to 278528 tid 1 2024-09-15T03:41:02.033 INFO:tasks.rados.rados.0.smithi105.stdout:18: write initial oid 18 2024-09-15T03:41:02.033 INFO:tasks.rados.rados.0.smithi105.stdout:18: seq_num 17 ranges {0=294912} 2024-09-15T03:41:02.035 INFO:tasks.rados.rados.0.smithi105.stdout:18: writing smithi10536496-18 from 0 to 294912 tid 1 2024-09-15T03:41:02.035 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 1 to smithi10536496-6 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 2 to smithi10536496-6 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 1 to smithi10536496-5 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 2 to smithi10536496-5 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 1 to smithi10536496-8 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 2 to smithi10536496-8 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 3 to smithi10536496-1 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 3 to smithi10536496-3 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-15T03:41:02.036 INFO:tasks.rados.rados.0.smithi105.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 1 to smithi10536496-10 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 2 to smithi10536496-10 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 1 to smithi10536496-9 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 1 to smithi10536496-7 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 2 to smithi10536496-9 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 3 to smithi10536496-6 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 2 to smithi10536496-7 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 1 to smithi10536496-12 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:1: done (15 left) 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:3: done (14 left) 2024-09-15T03:41:02.037 INFO:tasks.rados.rados.0.smithi105.stdout:6: done (13 left) 2024-09-15T03:41:02.038 INFO:tasks.rados.rados.0.smithi105.stdout:19: write initial oid 19 2024-09-15T03:41:02.038 INFO:tasks.rados.rados.0.smithi105.stdout:19: seq_num 18 ranges {0=311296} 2024-09-15T03:41:02.038 INFO:tasks.rados.rados.0.smithi105.stdout:19: writing smithi10536496-19 from 0 to 311296 tid 1 2024-09-15T03:41:02.038 INFO:tasks.rados.rados.0.smithi105.stdout:20: write initial oid 20 2024-09-15T03:41:02.038 INFO:tasks.rados.rados.0.smithi105.stdout:20: seq_num 19 ranges {0=327680} 2024-09-15T03:41:02.040 INFO:tasks.rados.rados.0.smithi105.stdout:20: writing smithi10536496-20 from 0 to 327680 tid 1 2024-09-15T03:41:02.040 INFO:tasks.rados.rados.0.smithi105.stdout:21: write initial oid 21 2024-09-15T03:41:02.040 INFO:tasks.rados.rados.0.smithi105.stdout:21: seq_num 20 ranges {0=344064} 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout:21: writing smithi10536496-21 from 0 to 344064 tid 1 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 3 to smithi10536496-8 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout:8: done (15 left) 2024-09-15T03:41:02.043 INFO:tasks.rados.rados.0.smithi105.stdout:22: write initial oid 22 2024-09-15T03:41:02.044 INFO:tasks.rados.rados.0.smithi105.stdout:22: seq_num 21 ranges {0=360448} 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout:22: writing smithi10536496-22 from 0 to 360448 tid 1 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 2 to smithi10536496-12 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 1 to smithi10536496-14 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 3 to smithi10536496-7 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-15T03:41:02.049 INFO:tasks.rados.rados.0.smithi105.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 3 to smithi10536496-5 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:5: done (15 left) 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:7: done (14 left) 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:23: write initial oid 23 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:23: seq_num 22 ranges {0=376832} 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:23: writing smithi10536496-23 from 0 to 376832 tid 1 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:24: write initial oid 24 2024-09-15T03:41:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:24: seq_num 23 ranges {0=393216} 2024-09-15T03:41:02.051 INFO:tasks.rados.rados.0.smithi105.stdout:24: writing smithi10536496-24 from 0 to 393216 tid 1 2024-09-15T03:41:02.051 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.051 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 2 to smithi10536496-14 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 1 to smithi10536496-11 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 2 to smithi10536496-11 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 1 to smithi10536496-15 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 3 to smithi10536496-14 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 1 to smithi10536496-13 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 2 to smithi10536496-13 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 1 to smithi10536496-16 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 2 to smithi10536496-16 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 3 to smithi10536496-11 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-15T03:41:02.052 INFO:tasks.rados.rados.0.smithi105.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-15T03:41:02.053 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 1 to smithi10536496-18 2024-09-15T03:41:02.053 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 1 to smithi10536496-17 2024-09-15T03:41:02.053 INFO:tasks.rados.rados.0.smithi105.stdout:11: done (15 left) 2024-09-15T03:41:02.053 INFO:tasks.rados.rados.0.smithi105.stdout:14: done (14 left) 2024-09-15T03:41:02.053 INFO:tasks.rados.rados.0.smithi105.stdout:25: write initial oid 25 2024-09-15T03:41:02.053 INFO:tasks.rados.rados.0.smithi105.stdout:25: seq_num 24 ranges {0=409600} 2024-09-15T03:41:02.054 INFO:tasks.rados.rados.0.smithi105.stdout:25: writing smithi10536496-25 from 0 to 409600 tid 1 2024-09-15T03:41:02.054 INFO:tasks.rados.rados.0.smithi105.stdout:26: write initial oid 26 2024-09-15T03:41:02.054 INFO:tasks.rados.rados.0.smithi105.stdout:26: seq_num 25 ranges {0=425984} 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:26: writing smithi10536496-26 from 0 to 425984 tid 1 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 2 to smithi10536496-17 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 3 to smithi10536496-9 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 1 to smithi10536496-19 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 2 to smithi10536496-19 2024-09-15T03:41:02.056 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 1 to smithi10536496-20 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 2 to smithi10536496-20 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 3 to smithi10536496-17 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:9: done (15 left) 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:17: done (14 left) 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:27: write initial oid 27 2024-09-15T03:41:02.057 INFO:tasks.rados.rados.0.smithi105.stdout:27: seq_num 26 ranges {0=442368} 2024-09-15T03:41:02.059 INFO:tasks.rados.rados.0.smithi105.stdout:27: writing smithi10536496-27 from 0 to 442368 tid 1 2024-09-15T03:41:02.059 INFO:tasks.rados.rados.0.smithi105.stdout:28: write initial oid 28 2024-09-15T03:41:02.059 INFO:tasks.rados.rados.0.smithi105.stdout:28: seq_num 27 ranges {0=458752} 2024-09-15T03:41:02.062 INFO:tasks.rados.rados.0.smithi105.stdout:28: writing smithi10536496-28 from 0 to 458752 tid 1 2024-09-15T03:41:02.062 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.062 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 3 to smithi10536496-20 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 4 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 3 to smithi10536496-19 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 1 to smithi10536496-21 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 2 to smithi10536496-21 2024-09-15T03:41:02.063 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 3 to smithi10536496-21 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 1 to smithi10536496-23 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 2 to smithi10536496-23 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 1 to smithi10536496-24 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 3 to smithi10536496-23 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 4 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 2 to smithi10536496-24 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 3 to smithi10536496-16 2024-09-15T03:41:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 3 to smithi10536496-24 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 1 to smithi10536496-25 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 2 to smithi10536496-25 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:16: done (15 left) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:19: done (14 left) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:20: done (13 left) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:21: done (12 left) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:23: done (11 left) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:24: done (10 left) 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:29: write initial oid 29 2024-09-15T03:41:02.065 INFO:tasks.rados.rados.0.smithi105.stdout:29: seq_num 28 ranges {0=475136} 2024-09-15T03:41:02.066 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:02.062+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:02.066 INFO:tasks.rados.rados.0.smithi105.stdout:29: writing smithi10536496-29 from 0 to 475136 tid 1 2024-09-15T03:41:02.067 INFO:tasks.rados.rados.0.smithi105.stdout:30: write initial oid 30 2024-09-15T03:41:02.067 INFO:tasks.rados.rados.0.smithi105.stdout:30: seq_num 29 ranges {0=491520} 2024-09-15T03:41:02.068 INFO:tasks.rados.rados.0.smithi105.stdout:30: writing smithi10536496-30 from 0 to 491520 tid 1 2024-09-15T03:41:02.068 INFO:tasks.rados.rados.0.smithi105.stdout:31: write initial oid 31 2024-09-15T03:41:02.068 INFO:tasks.rados.rados.0.smithi105.stdout:31: seq_num 30 ranges {0=507904} 2024-09-15T03:41:02.071 INFO:tasks.rados.rados.0.smithi105.stdout:31: writing smithi10536496-31 from 0 to 507904 tid 1 2024-09-15T03:41:02.071 INFO:tasks.rados.rados.0.smithi105.stdout:32: write initial oid 32 2024-09-15T03:41:02.071 INFO:tasks.rados.rados.0.smithi105.stdout:32: seq_num 31 ranges {0=524288} 2024-09-15T03:41:02.074 INFO:tasks.rados.rados.0.smithi105.stdout:32: writing smithi10536496-32 from 0 to 524288 tid 1 2024-09-15T03:41:02.074 INFO:tasks.rados.rados.0.smithi105.stdout:33: write initial oid 33 2024-09-15T03:41:02.074 INFO:tasks.rados.rados.0.smithi105.stdout:33: seq_num 32 ranges {0=540672} 2024-09-15T03:41:02.077 INFO:tasks.rados.rados.0.smithi105.stdout:33: writing smithi10536496-33 from 0 to 540672 tid 1 2024-09-15T03:41:02.077 INFO:tasks.rados.rados.0.smithi105.stdout:34: write initial oid 34 2024-09-15T03:41:02.078 INFO:tasks.rados.rados.0.smithi105.stdout:34: seq_num 33 ranges {0=557056} 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout:34: writing smithi10536496-34 from 0 to 557056 tid 1 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 3 to smithi10536496-25 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout:25: done (15 left) 2024-09-15T03:41:02.082 INFO:tasks.rados.rados.0.smithi105.stdout:35: write initial oid 35 2024-09-15T03:41:02.083 INFO:tasks.rados.rados.0.smithi105.stdout:35: seq_num 34 ranges {0=573440} 2024-09-15T03:41:02.087 INFO:tasks.rados.rados.0.smithi105.stdout:35: writing smithi10536496-35 from 0 to 573440 tid 1 2024-09-15T03:41:02.087 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.087 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 1 to smithi10536496-26 2024-09-15T03:41:02.087 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 1 to smithi10536496-27 2024-09-15T03:41:02.087 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 2 to smithi10536496-27 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 1 to smithi10536496-28 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 3 to smithi10536496-12 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 1 to smithi10536496-22 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 2 to smithi10536496-28 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 3 to smithi10536496-27 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 1 to smithi10536496-30 2024-09-15T03:41:02.088 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 3 to smithi10536496-28 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 2 to smithi10536496-30 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 1 to smithi10536496-32 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 3 to smithi10536496-13 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 1 to smithi10536496-31 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 3 to smithi10536496-10 2024-09-15T03:41:02.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 2 to smithi10536496-26 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 2 to smithi10536496-31 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 3 to smithi10536496-30 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 1 to smithi10536496-33 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 2 to smithi10536496-32 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:10: done (15 left) 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:12: done (14 left) 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:13: done (13 left) 2024-09-15T03:41:02.090 INFO:tasks.rados.rados.0.smithi105.stdout:27: done (12 left) 2024-09-15T03:41:02.091 INFO:tasks.rados.rados.0.smithi105.stdout:28: done (11 left) 2024-09-15T03:41:02.091 INFO:tasks.rados.rados.0.smithi105.stdout:30: done (10 left) 2024-09-15T03:41:02.091 INFO:tasks.rados.rados.0.smithi105.stdout:36: write initial oid 36 2024-09-15T03:41:02.091 INFO:tasks.rados.rados.0.smithi105.stdout:36: seq_num 35 ranges {0=589824} 2024-09-15T03:41:02.091 INFO:tasks.rados.rados.0.smithi105.stdout:36: writing smithi10536496-36 from 0 to 589824 tid 1 2024-09-15T03:41:02.092 INFO:tasks.rados.rados.0.smithi105.stdout:37: write initial oid 37 2024-09-15T03:41:02.092 INFO:tasks.rados.rados.0.smithi105.stdout:37: seq_num 36 ranges {0=606208} 2024-09-15T03:41:02.093 INFO:tasks.rados.rados.0.smithi105.stdout:37: writing smithi10536496-37 from 0 to 606208 tid 1 2024-09-15T03:41:02.093 INFO:tasks.rados.rados.0.smithi105.stdout:38: write initial oid 38 2024-09-15T03:41:02.093 INFO:tasks.rados.rados.0.smithi105.stdout:38: seq_num 37 ranges {0=622592} 2024-09-15T03:41:02.097 INFO:tasks.rados.rados.0.smithi105.stdout:38: writing smithi10536496-38 from 0 to 622592 tid 1 2024-09-15T03:41:02.097 INFO:tasks.rados.rados.0.smithi105.stdout:39: write initial oid 39 2024-09-15T03:41:02.097 INFO:tasks.rados.rados.0.smithi105.stdout:39: seq_num 38 ranges {0=638976} 2024-09-15T03:41:02.101 INFO:tasks.rados.rados.0.smithi105.stdout:39: writing smithi10536496-39 from 0 to 638976 tid 1 2024-09-15T03:41:02.101 INFO:tasks.rados.rados.0.smithi105.stdout:40: write initial oid 40 2024-09-15T03:41:02.101 INFO:tasks.rados.rados.0.smithi105.stdout:40: seq_num 39 ranges {0=671744} 2024-09-15T03:41:02.106 INFO:tasks.rados.rados.0.smithi105.stdout:40: writing smithi10536496-40 from 0 to 671744 tid 1 2024-09-15T03:41:02.106 INFO:tasks.rados.rados.0.smithi105.stdout:41: write initial oid 41 2024-09-15T03:41:02.106 INFO:tasks.rados.rados.0.smithi105.stdout:41: seq_num 40 ranges {0=688128} 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:41: writing smithi10536496-41 from 0 to 688128 tid 1 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 2 to smithi10536496-33 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 3 to smithi10536496-26 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 3 to smithi10536496-32 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 2 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-15T03:41:02.111 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 1 to smithi10536496-34 2024-09-15T03:41:02.112 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 1 to smithi10536496-36 2024-09-15T03:41:02.112 INFO:tasks.rados.rados.0.smithi105.stdout:26: done (15 left) 2024-09-15T03:41:02.112 INFO:tasks.rados.rados.0.smithi105.stdout:32: done (14 left) 2024-09-15T03:41:02.112 INFO:tasks.rados.rados.0.smithi105.stdout:42: write initial oid 42 2024-09-15T03:41:02.112 INFO:tasks.rados.rados.0.smithi105.stdout:42: seq_num 41 ranges {0=704512} 2024-09-15T03:41:02.114 INFO:tasks.rados.rados.0.smithi105.stdout:42: writing smithi10536496-42 from 0 to 704512 tid 1 2024-09-15T03:41:02.114 INFO:tasks.rados.rados.0.smithi105.stdout:43: write initial oid 43 2024-09-15T03:41:02.115 INFO:tasks.rados.rados.0.smithi105.stdout:43: seq_num 42 ranges {0=720896} 2024-09-15T03:41:02.119 INFO:tasks.rados.rados.0.smithi105.stdout:43: writing smithi10536496-43 from 0 to 720896 tid 1 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 2 to smithi10536496-36 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 1 to smithi10536496-35 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 2 to smithi10536496-35 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 3 to smithi10536496-36 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 1 to smithi10536496-38 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 2 to smithi10536496-38 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 1 to smithi10536496-37 2024-09-15T03:41:02.120 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 2 to smithi10536496-37 2024-09-15T03:41:02.121 INFO:tasks.rados.rados.0.smithi105.stdout:36: done (15 left) 2024-09-15T03:41:02.121 INFO:tasks.rados.rados.0.smithi105.stdout:44: write initial oid 44 2024-09-15T03:41:02.121 INFO:tasks.rados.rados.0.smithi105.stdout:44: seq_num 43 ranges {0=737280} 2024-09-15T03:41:02.125 INFO:tasks.rados.rados.0.smithi105.stdout:44: writing smithi10536496-44 from 0 to 737280 tid 1 2024-09-15T03:41:02.125 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.125 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 1 to smithi10536496-39 2024-09-15T03:41:02.125 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 3 to smithi10536496-35 2024-09-15T03:41:02.125 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 3 to smithi10536496-38 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 2 to smithi10536496-39 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 3 to smithi10536496-39 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:35: done (15 left) 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:38: done (14 left) 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:39: done (13 left) 2024-09-15T03:41:02.126 INFO:tasks.rados.rados.0.smithi105.stdout:45: write initial oid 45 2024-09-15T03:41:02.127 INFO:tasks.rados.rados.0.smithi105.stdout:45: seq_num 44 ranges {0=753664} 2024-09-15T03:41:02.129 INFO:tasks.rados.rados.0.smithi105.stdout:45: writing smithi10536496-45 from 0 to 753664 tid 1 2024-09-15T03:41:02.129 INFO:tasks.rados.rados.0.smithi105.stdout:46: write initial oid 46 2024-09-15T03:41:02.129 INFO:tasks.rados.rados.0.smithi105.stdout:46: seq_num 45 ranges {0=770048} 2024-09-15T03:41:02.131 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:02.130+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:46: writing smithi10536496-46 from 0 to 770048 tid 1 2024-09-15T03:41:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:47: write initial oid 47 2024-09-15T03:41:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:47: seq_num 46 ranges {0=786432} 2024-09-15T03:41:02.140 INFO:tasks.rados.rados.0.smithi105.stdout:47: writing smithi10536496-47 from 0 to 786432 tid 1 2024-09-15T03:41:02.140 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.140 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 3 to smithi10536496-31 2024-09-15T03:41:02.140 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 2 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-15T03:41:02.140 INFO:tasks.rados.rados.0.smithi105.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-15T03:41:02.140 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 3 to smithi10536496-33 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 1 to smithi10536496-41 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 2 to smithi10536496-22 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 2 to smithi10536496-41 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 1 to smithi10536496-42 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 2 to smithi10536496-42 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:31: done (15 left) 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:33: done (14 left) 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:48: write initial oid 48 2024-09-15T03:41:02.141 INFO:tasks.rados.rados.0.smithi105.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-15T03:41:02.142 INFO:tasks.rados.rados.0.smithi105.stdout:48: writing smithi10536496-48 from 0 to 425984 tid 1 2024-09-15T03:41:02.144 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:02.144 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:02.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:02.144 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:02.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.145 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.6s5 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:02.145 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018298+0000", 2024-09-15T03:41:02.145 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.123541752, 2024-09-15T03:41:02.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00384199, 2024-09-15T03:41:02.145 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018298+0000", 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018298+0000", 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018298+0000", 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8099999999999999e-07 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018374+0000", 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6453999999999997e-05 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018407+0000", 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2379999999999998e-05 2024-09-15T03:41:02.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018416+0000", 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1919999999999994e-06 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018510+0000", 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4333999999999995e-05 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018519+0000", 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3000000000000002e-06 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.022121+0000", 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003602007 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.022140+0000", 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9042000000000002e-05 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.10880092800000001, 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0090780140000000006, 2024-09-15T03:41:02.150 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0100000000000005e-07 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5333e-05 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041513+0000", 2024-09-15T03:41:02.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084585600000000004 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041517+0000", 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.318e-06 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041556+0000", 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8426000000000001e-05 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041563+0000", 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6930000000000007e-06 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042090+0000", 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052617600000000005 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042117+0000", 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6906999999999998e-05 2024-09-15T03:41:02.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=6, to_read={3:5db82ad9:::smithi10536496-11:head=0,4096,0}, subchunks={3:5db82ad9:::smithi10536496-11:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.041492+0000", 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.10034712, 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052439380000000001, 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041492+0000", 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041492+0000", 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041493+0000", 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2900000000000002e-07 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041500+0000", 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3990000000000001e-06 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046417+0000", 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0049176109999999997 2024-09-15T03:41:02.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046425+0000", 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8050000000000007e-06 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046478+0000", 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2729000000000003e-05 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046736+0000", 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025856499999999999 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.094864770000000001, 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.159 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=9, reqid=client.4562.0:76, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.057351+0000", 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.084488302000000001, 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039463329999999998, 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.163 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057351+0000", 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057351+0000", 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057352+0000", 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8700000000000005e-07 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057464+0000", 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011213 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058644+0000", 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011795480000000001 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058651+0000", 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.277e-06 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058693+0000", 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1689e-05 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058703+0000", 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061274+0000", 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025718030000000001 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061298+0000", 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3057e-05 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.083235871000000003, 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:02.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:41:02.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.172 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.6s5 35/32 ECSubWrite(tid=4, reqid=client.4562.0:79, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.060686+0000", 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.081153115999999997, 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003736212, 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.060686+0000", 2024-09-15T03:41:02.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.060686+0000", 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.060687+0000", 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3799999999999996e-07 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061060+0000", 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00037240500000000002 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061095+0000", 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5287999999999998e-05 2024-09-15T03:41:02.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061102+0000", 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4939999999999997e-06 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061148+0000", 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5649000000000003e-05 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.061155+0000", 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1550000000000001e-06 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.064394+0000", 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0032385349999999999 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.064423+0000", 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8948000000000001e-05 2024-09-15T03:41:02.177 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:91 3.ds0 3:baae61d4:::smithi10536496-31:head [append 0~507904,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.066016460999999999, 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0073884809999999997, 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 91 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:02.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075824+0000", 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3069999999999999e-06 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076181+0000", 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00035651600000000003 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076196+0000", 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5308e-05 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076206+0000", 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4469999999999995e-06 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076297+0000", 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1287999999999997e-05 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076465+0000", 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016844599999999999 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.077478+0000", 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010127140000000001 2024-09-15T03:41:02.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080498+0000", 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030201759999999999 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083146+0000", 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026477100000000002 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083212+0000", 2024-09-15T03:41:02.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5568999999999995e-05 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:92 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.065617492999999999, 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078039399999999997, 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 92 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076223+0000", 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076226+0000", 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6210000000000001e-06 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076234+0000", 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2740000000000002e-06 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076238+0000", 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4329999999999997e-06 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078113+0000", 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018749769999999999 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078219+0000", 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106016 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078626+0000", 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040713400000000002 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080519+0000", 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001892712 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083957+0000", 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034382610000000002 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084026+0000", 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8678999999999998e-05 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.189 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.065592370999999997, 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.190 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:41:02.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.055766622000000002, 2024-09-15T03:41:02.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:41:02.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=13, reqid=client.4562.0:121, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.111820+0000", 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.030019820999999999, 2024-09-15T03:41:02.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0038574059999999999, 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111820+0000", 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111820+0000", 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111821+0000", 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.531e-06 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111952+0000", 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112963+0000", 2024-09-15T03:41:02.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010109310000000001 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112969+0000", 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6270000000000002e-06 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.113015+0000", 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5728e-05 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.113024+0000", 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8659999999999995e-06 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115655+0000", 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002631516 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115677+0000", 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1800000000000001e-05 2024-09-15T03:41:02.203 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=14, reqid=client.4562.0:122, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.112925+0000", 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.028914180000000001, 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062104780000000002, 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112925+0000", 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112925+0000", 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112926+0000", 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.54e-07 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112942+0000", 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118742+0000", 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0057993089999999999 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118746+0000", 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4630000000000003e-06 2024-09-15T03:41:02.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118786+0000", 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0299999999999997e-05 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118803+0000", 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6532e-05 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.119119+0000", 2024-09-15T03:41:02.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000316142 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.119136+0000", 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6671999999999999e-05 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:124 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~704512,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.208 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.114765+0000", 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.027074274999999998, 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0070682460000000003, 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 124 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.114765+0000", 2024-09-15T03:41:02.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.114765+0000", 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.114766+0000", 2024-09-15T03:41:02.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2759999999999999e-06 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115094+0000", 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032749199999999997 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115106+0000", 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2058999999999999e-05 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115111+0000", 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4280000000000004e-06 2024-09-15T03:41:02.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4959000000000001e-05 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115311+0000", 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015452599999999999 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116262+0000", 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095060800000000003 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118414+0000", 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002152798 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121758+0000", 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033438949999999999 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121833+0000", 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5204999999999996e-05 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:125 3.ds0 3:b9f64cc8:::smithi10536496-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.026719697000000001, 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071775770000000001, 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.215 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 125 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8000000000000001e-07 2024-09-15T03:41:02.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115122+0000", 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2620000000000001e-06 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115128+0000", 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5729999999999998e-06 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115131+0000", 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6630000000000002e-06 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116937+0000", 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018060249999999999 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116989+0000", 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1934000000000001e-05 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117215+0000", 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022591199999999999 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118438+0000", 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012237699999999999 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122235+0000", 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037965799999999999 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122297+0000", 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2477999999999994e-05 2024-09-15T03:41:02.221 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.026689589, 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.222 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:41:02.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:41:02.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=5, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.118729+0000", 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.023110691999999999, 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039120129999999998, 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118729+0000", 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118729+0000", 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118729+0000", 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118730+0000", 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1689999999999999e-06 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122123+0000", 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033925270000000002 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122127+0000", 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.861e-06 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122172+0000", 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5318999999999998e-05 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122641+0000", 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046898099999999999 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.231 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.3s1 35/32 ECSubWrite(tid=120, reqid=client.4562.0:127, at_version=35'3, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.121189+0000", 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.020650017999999999, 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0047254030000000004, 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121189+0000", 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121189+0000", 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121190+0000", 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8699999999999995e-07 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122048+0000", 2024-09-15T03:41:02.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00085789600000000003 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122100+0000", 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2441000000000002e-05 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122107+0000", 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5649999999999998e-06 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122155+0000", 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8029999999999999e-05 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122163+0000", 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4819999999999994e-06 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125890+0000", 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037268510000000002 2024-09-15T03:41:02.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125915+0000", 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4550999999999999e-05 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=15, reqid=client.4562.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.122066+0000", 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.019773474999999999, 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0038269699999999999, 2024-09-15T03:41:02.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122066+0000", 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122066+0000", 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122067+0000", 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5000000000000002e-07 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122074+0000", 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0419999999999997e-06 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122138+0000", 2024-09-15T03:41:02.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4596000000000003e-05 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122142+0000", 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6780000000000002e-06 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122420+0000", 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000277667 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122431+0000", 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125871+0000", 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034407040000000002 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125893+0000", 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1741000000000001e-05 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.240 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.3s1 35/32 ECSubWrite(tid=125, reqid=client.4562.0:133, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.131894+0000", 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0099458350000000001, 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039032390000000002, 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.131894+0000", 2024-09-15T03:41:02.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.131894+0000", 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.131894+0000", 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8899999999999998e-07 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.132251+0000", 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00035632300000000002 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.132269+0000", 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7832000000000001e-05 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.132275+0000", 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8079999999999999e-06 2024-09-15T03:41:02.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.132318+0000", 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3065999999999997e-05 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.132328+0000", 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6020000000000006e-06 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.135769+0000", 2024-09-15T03:41:02.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003441239 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.135797+0000", 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.758e-05 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.245 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:48: writing smithi10536496-48 from 425984 to 802816 tid 2 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:49: write initial oid 49 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:49: writing smithi10536496-49 from 0 to 442368 tid 1 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:49: writing smithi10536496-49 from 442368 to 819200 tid 2 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 1 to smithi10536496-40 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 1 to smithi10536496-29 2024-09-15T03:41:02.249 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 3 to smithi10536496-41 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 8 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 3 to smithi10536496-42 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 4 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 1 to smithi10536496-43 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 2 to smithi10536496-43 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 2 to smithi10536496-18 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 3 to smithi10536496-43 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 4 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 1 to smithi10536496-44 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 2 to smithi10536496-44 2024-09-15T03:41:02.250 INFO:tasks.rados.rados.0.smithi105.stdout:41: done (15 left) 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:42: done (14 left) 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:43: done (13 left) 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:50: write initial oid 50 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:50: writing smithi10536496-50 from 0 to 458752 tid 1 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:50: writing smithi10536496-50 from 458752 to 835584 tid 2 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:51: copy_from oid 16 from oid 41 current snap is 0 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 3 to smithi10536496-44 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 1 to smithi10536496-46 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:44: done (14 left) 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:52: read oid 39 snap -1 2024-09-15T03:41:02.251 INFO:tasks.rados.rados.0.smithi105.stdout:52: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 8 current snap is 0 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 1 to smithi10536496-45 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 2 to smithi10536496-46 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 2 to smithi10536496-45 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 3 to smithi10536496-45 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 2 to smithi10536496-34 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 3 to smithi10536496-37 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 8 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 3 to smithi10536496-46 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 8 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-15T03:41:02.252 INFO:tasks.rados.rados.0.smithi105.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 1 to smithi10536496-47 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 1 to smithi10536496-48 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 3 to smithi10536496-18 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 2 to smithi10536496-48 2024-09-15T03:41:02.253 INFO:tasks.rados.rados.0.smithi105.stdout:53: seq_num 50 ranges {131072=475136,606208=376832} 2024-09-15T03:41:02.254 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:02.255 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:02.248+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:02.258 INFO:tasks.rados.rados.0.smithi105.stdout:53: writing smithi10536496-8 from 131072 to 606208 tid 1 2024-09-15T03:41:02.258 INFO:tasks.rados.rados.0.smithi105.stdout:53: writing smithi10536496-8 from 606208 to 983040 tid 2 2024-09-15T03:41:02.258 INFO:tasks.rados.rados.0.smithi105.stdout:18: done (15 left) 2024-09-15T03:41:02.258 INFO:tasks.rados.rados.0.smithi105.stdout:37: done (14 left) 2024-09-15T03:41:02.258 INFO:tasks.rados.rados.0.smithi105.stdout:45: done (13 left) 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:46: done (12 left) 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:54: setattr oid 14 current snap is 0 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 2 to smithi10536496-40 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 1 to smithi10536496-49 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 2 to smithi10536496-49 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 3 to smithi10536496-49 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 1 to smithi10536496-50 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:52: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 2 to smithi10536496-50 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 3 to smithi10536496-40 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-15T03:41:02.259 INFO:tasks.rados.rados.0.smithi105.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 2 to smithi10536496-29 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 3 to smithi10536496-48 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 2 to smithi10536496-15 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 3 to smithi10536496-22 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 3 to smithi10536496-15 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 3 to smithi10536496-29 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 10 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-15T03:41:02.260 INFO:tasks.rados.rados.0.smithi105.stdout:51: finishing copy_from racing read to smithi10536496-16 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 4 to smithi10536496-48 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 11 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:53: finishing write tid 1 to smithi10536496-8 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:15: done (12 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:22: done (11 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:29: done (10 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:40: done (9 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:48: done (8 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:52: done (7 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:55: rmattr oid 33 current snap is 0 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:55: done (7 left) 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 2 current snap is 0 2024-09-15T03:41:02.261 INFO:tasks.rados.rados.0.smithi105.stdout:56: seq_num 51 ranges {32768=491520,524288=376832} 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:56: writing smithi10536496-2 from 32768 to 524288 tid 1 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:56: writing smithi10536496-2 from 524288 to 901120 tid 2 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:57: rmattr oid 13 current snap is 0 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:57: done (8 left) 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:58: delete oid 45 current snap is 0 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 2 to smithi10536496-47 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 3 to smithi10536496-34 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 3 to smithi10536496-50 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:34: done (8 left) 2024-09-15T03:41:02.262 INFO:tasks.rados.rados.0.smithi105.stdout:54: done (7 left) 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:58: done (6 left) 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 13 current snap is 0 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:59: seq_num 53 ranges {212992=524288,737280=376832} 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:59: writing smithi10536496-13 from 212992 to 737280 tid 1 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:59: writing smithi10536496-13 from 737280 to 1114112 tid 2 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:60: read oid 40 snap -1 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:60: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:61: delete oid 7 current snap is 0 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:56: finishing write tid 1 to smithi10536496-2 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:56: finishing write tid 2 to smithi10536496-2 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 4 to smithi10536496-50 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 4 to smithi10536496-49 2024-09-15T03:41:02.263 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 3 to smithi10536496-47 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:59: finishing write tid 1 to smithi10536496-13 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:59: finishing write tid 2 to smithi10536496-13 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:53: finishing write tid 2 to smithi10536496-8 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:51: finishing copy_from to smithi10536496-16 2024-09-15T03:41:02.264 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 8 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-15T03:41:02.265 INFO:tasks.rados.rados.0.smithi105.stdout:60: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-15T03:41:02.286 INFO:tasks.rados.rados.0.smithi105.stdout:56: finishing write tid 3 to smithi10536496-2 2024-09-15T03:41:02.287 INFO:tasks.rados.rados.0.smithi105.stdout:56: finishing write tid 4 to smithi10536496-2 2024-09-15T03:41:02.287 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-15T03:41:02.287 INFO:tasks.rados.rados.0.smithi105.stdout:56: left oid 2 (ObjNum 51 snap 0 seq_num 51) 2024-09-15T03:41:02.349 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:02.348+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:02.361 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:171 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.16746164799999999, 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16748581000000001, 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 171 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193987+0000", 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193989+0000", 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6410000000000002e-06 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193994+0000", 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193996+0000", 2024-09-15T03:41:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00136275 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195404+0000", 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5433999999999999e-05 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195711+0000", 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030663100000000003 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196156+0000", 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044516899999999998 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:172 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.16743033500000001, 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16750142500000001, 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "waiting for rw locks", 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 172 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194018+0000", 2024-09-15T03:41:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5699999999999998e-07 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194020+0000", 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194026+0000", 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194028+0000", 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.475e-06 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195860+0000", 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018314609999999999 2024-09-15T03:41:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195941+0000", 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1464999999999993e-05 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 2 2024-09-15T03:41:02.371 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:02.372 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:02.451 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:02.450+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:02.467 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:02.510 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:02.510 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:02.510 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:02.510 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:02.510 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:02.518 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:02.562 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:02.561+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:02.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:02.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:02.617 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:02.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:02.620 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:02.620 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:02.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:02.621 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:02.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.621 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:16 3.cs0 3:37f43638:::smithi10536496-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:02.621 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018343+0000", 2024-09-15T03:41:02.622 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59723825600000002, 2024-09-15T03:41:02.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013177902999999999, 2024-09-15T03:41:02.622 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.622 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.623 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.623 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 16 2024-09-15T03:41:02.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.624 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018343+0000", 2024-09-15T03:41:02.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018343+0000", 2024-09-15T03:41:02.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018344+0000", 2024-09-15T03:41:02.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8449999999999999e-06 2024-09-15T03:41:02.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018432+0000", 2024-09-15T03:41:02.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7736000000000002e-05 2024-09-15T03:41:02.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018444+0000", 2024-09-15T03:41:02.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.221e-05 2024-09-15T03:41:02.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018451+0000", 2024-09-15T03:41:02.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0029999999999997e-06 2024-09-15T03:41:02.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018503+0000", 2024-09-15T03:41:02.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1323000000000003e-05 2024-09-15T03:41:02.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018658+0000", 2024-09-15T03:41:02.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015490999999999999 2024-09-15T03:41:02.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019317+0000", 2024-09-15T03:41:02.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00065981999999999998 2024-09-15T03:41:02.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.022787+0000", 2024-09-15T03:41:02.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034691050000000001 2024-09-15T03:41:02.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031454+0000", 2024-09-15T03:41:02.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0086671630000000003 2024-09-15T03:41:02.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031520+0000", 2024-09-15T03:41:02.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6787999999999998e-05 2024-09-15T03:41:02.635 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.635 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:17 3.cs0 3:37f43638:::smithi10536496-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:02.636 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018661+0000", 2024-09-15T03:41:02.636 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59692025500000001, 2024-09-15T03:41:02.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013020116999999999, 2024-09-15T03:41:02.637 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.637 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.638 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 17 2024-09-15T03:41:02.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.638 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018661+0000", 2024-09-15T03:41:02.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018661+0000", 2024-09-15T03:41:02.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018661+0000", 2024-09-15T03:41:02.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3900000000000004e-07 2024-09-15T03:41:02.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018666+0000", 2024-09-15T03:41:02.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6639999999999997e-06 2024-09-15T03:41:02.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018675+0000", 2024-09-15T03:41:02.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1439999999999992e-06 2024-09-15T03:41:02.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018681+0000", 2024-09-15T03:41:02.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8629999999999996e-06 2024-09-15T03:41:02.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019669+0000", 2024-09-15T03:41:02.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00098753800000000004 2024-09-15T03:41:02.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019782+0000", 2024-09-15T03:41:02.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113711 2024-09-15T03:41:02.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.020368+0000", 2024-09-15T03:41:02.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00058550699999999998 2024-09-15T03:41:02.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.022846+0000", 2024-09-15T03:41:02.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024779659999999998 2024-09-15T03:41:02.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031615+0000", 2024-09-15T03:41:02.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087689610000000005 2024-09-15T03:41:02.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031681+0000", 2024-09-15T03:41:02.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6024000000000003e-05 2024-09-15T03:41:02.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.650 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.651 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:18 3.cs0 3:37f43638:::smithi10536496-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:02.651 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:02.651 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59687212700000003, 2024-09-15T03:41:02.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016101061999999999, 2024-09-15T03:41:02.651 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.652 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 18 2024-09-15T03:41:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:02.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:02.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5000000000000003e-07 2024-09-15T03:41:02.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018712+0000", 2024-09-15T03:41:02.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9440000000000001e-06 2024-09-15T03:41:02.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018720+0000", 2024-09-15T03:41:02.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3980000000000002e-06 2024-09-15T03:41:02.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018731+0000", 2024-09-15T03:41:02.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1385e-05 2024-09-15T03:41:02.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.020559+0000", 2024-09-15T03:41:02.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018280589999999999 2024-09-15T03:41:02.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.020635+0000", 2024-09-15T03:41:02.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6260000000000005e-05 2024-09-15T03:41:02.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031717+0000", 2024-09-15T03:41:02.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011081573000000001 2024-09-15T03:41:02.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031804+0000", 2024-09-15T03:41:02.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7610999999999999e-05 2024-09-15T03:41:02.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.034810+0000", 2024-09-15T03:41:02.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030052820000000002 2024-09-15T03:41:02.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.663 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:28 3.fs0 3:fff44eab:::smithi10536496-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018736+0000", 2024-09-15T03:41:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59684495800000004, 2024-09-15T03:41:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014367586, 2024-09-15T03:41:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 28 2024-09-15T03:41:02.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.666 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018736+0000", 2024-09-15T03:41:02.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018736+0000", 2024-09-15T03:41:02.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018736+0000", 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4200000000000001e-07 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018825+0000", 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9086999999999996e-05 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018832+0000", 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0809999999999996e-06 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018837+0000", 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.617e-06 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018880+0000", 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2698999999999999e-05 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op must wait for map", 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018885+0000", 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6509999999999998e-06 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.023550+0000", 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046647959999999997 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.023667+0000", 2024-09-15T03:41:02.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116589 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024423+0000", 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00075630100000000002 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026582+0000", 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0021593039999999999 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033045+0000", 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064622910000000002 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033103+0000", 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8727999999999997e-05 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.674 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:29 3.fs0 3:fff44eab:::smithi10536496-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018925+0000", 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59665563799999999, 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014361852, 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 29 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018925+0000", 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018925+0000", 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018926+0000", 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5799999999999999e-07 2024-09-15T03:41:02.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018929+0000", 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.512e-06 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018936+0000", 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0010000000000004e-06 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018941+0000", 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3669999999999999e-06 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019155+0000", 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000214472 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019170+0000", 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5282e-05 2024-09-15T03:41:02.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024827+0000", 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056570129999999998 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024926+0000", 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8999999999999994e-05 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025410+0000", 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00048357100000000002 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026916+0000", 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001506435 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033226+0000", 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063094070000000004 2024-09-15T03:41:02.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033287+0000", 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1134000000000001e-05 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:30 3.fs0 3:fff44eab:::smithi10536496-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59639687799999996, 2024-09-15T03:41:02.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062015339000000003, 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 30 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019185+0000", 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999995e-07 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019188+0000", 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6160000000000002e-06 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019196+0000", 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8699999999999992e-06 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019201+0000", 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019234+0000", 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3402000000000003e-05 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019239+0000", 2024-09-15T03:41:02.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1450000000000001e-06 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025625+0000", 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063864280000000004 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025701+0000", 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6044999999999994e-05 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033568+0000", 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078674130000000002 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033650+0000", 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1404999999999997e-05 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.081199+0000", 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047549438999999999 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:37 3.cs0 3:36fe1c1b:::smithi10536496-13:head [append 0~212992,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.024171+0000", 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59140969899999996, 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011374242, 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 37 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024171+0000", 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024171+0000", 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024173+0000", 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.545e-06 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024306+0000", 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013311400000000001 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024316+0000", 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0336999999999999e-05 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024322+0000", 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1829999999999999e-06 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024363+0000", 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1003e-05 2024-09-15T03:41:02.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024492+0000", 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129085 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025545+0000", 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00105221 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026960+0000", 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001415216 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035484+0000", 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085239830000000006 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035545+0000", 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1566e-05 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.693 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:38 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.024800+0000", 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59078045199999996, 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019995407999999999, 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.694 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 38 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024800+0000", 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024800+0000", 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024801+0000", 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000002e-07 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024806+0000", 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5879999999999999e-06 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024815+0000", 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1390000000000001e-06 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024820+0000", 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025938+0000", 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00111758 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026026+0000", 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8487999999999997e-05 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026351+0000", 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032436800000000001 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026964+0000", 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00061353500000000001 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044742+0000", 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017777830000000001 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044796+0000", 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3792000000000003e-05 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:39 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.024829+0000", 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.59075180800000004, 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022622278999999999, 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 39 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024829+0000", 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024829+0000", 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024830+0000", 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9500000000000003e-07 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024832+0000", 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.977e-06 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024839+0000", 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7129999999999999e-06 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024843+0000", 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.76e-06 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026689+0000", 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018464099999999999 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026768+0000", 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8566000000000006e-05 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044854+0000", 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018086008000000001 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044950+0000", 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6056000000000003e-05 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047451+0000", 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025012939999999998 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.705 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:52 3.cs0 3:35272f51:::smithi10536496-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035444+0000", 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.58013695600000004, 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0098594800000000003, 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.706 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 52 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035444+0000", 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035444+0000", 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035445+0000", 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4050000000000001e-06 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035648+0000", 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020308300000000001 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035658+0000", 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9070000000000002e-06 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035664+0000", 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4709999999999998e-06 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035697+0000", 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3439999999999998e-05 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035808+0000", 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110659 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036252+0000", 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044422199999999998 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037385+0000", 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001132582 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.045240+0000", 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007855094 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.045303+0000", 2024-09-15T03:41:02.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3616999999999998e-05 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:53 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.57990746599999998, 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.089580073999999996, 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.712 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 53 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035674+0000", 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7000000000000005e-07 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035677+0000", 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4589999999999998e-06 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035684+0000", 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2659999999999998e-06 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035694+0000", 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8849999999999994e-06 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:02.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096889100000000004 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036718+0000", 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5436000000000003e-05 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037156+0000", 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043800399999999998 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037610+0000", 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00045398200000000002 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125179+0000", 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.087569006000000005 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125253+0000", 2024-09-15T03:41:02.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4574999999999997e-05 2024-09-15T03:41:02.718 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.718 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.718 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:54 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.57988113200000002, 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093566412000000002, 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 54 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.719 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035702+0000", 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1490000000000001e-06 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035709+0000", 2024-09-15T03:41:02.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6630000000000004e-06 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035713+0000", 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7780000000000001e-06 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037308+0000", 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015947769999999999 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037380+0000", 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2293999999999996e-05 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125568+0000", 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.088188213000000001 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125642+0000", 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4215999999999996e-05 2024-09-15T03:41:02.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.129266+0000", 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003623799 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as6 35/32 ECSubWrite(tid=9, reqid=client.4562.0:76, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.057422+0000", 2024-09-15T03:41:02.724 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.55815928999999997, 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067407730000000003, 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057422+0000", 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057422+0000", 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057422+0000", 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057492+0000", 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9305000000000001e-05 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057534+0000", 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2262000000000003e-05 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057540+0000", 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.579e-06 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057587+0000", 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6836999999999999e-05 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057596+0000", 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9670000000000007e-06 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.064134+0000", 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0065377129999999997 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.064162+0000", 2024-09-15T03:41:02.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8419e-05 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:98 3.fs0 3:f397533b:::smithi10536496-33:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.078294+0000", 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.53728664299999995, 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067573879999999996, 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.729 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 98 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078294+0000", 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078294+0000", 2024-09-15T03:41:02.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078295+0000", 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1999999999999999e-07 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078300+0000", 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6319999999999999e-06 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078307+0000", 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9200000000000004e-06 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078312+0000", 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4479999999999996e-06 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080304+0000", 2024-09-15T03:41:02.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019924700000000001 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080392+0000", 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7458000000000002e-05 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080795+0000", 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040269399999999999 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083252+0000", 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002457487 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084990+0000", 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017382470000000001 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.085052+0000", 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1311999999999998e-05 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.734 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:99 3.fs0 3:f397533b:::smithi10536496-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.078495+0000", 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.53708552399999998, 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088303749999999997, 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.735 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 99 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078495+0000", 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078495+0000", 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078496+0000", 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4300000000000003e-07 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078499+0000", 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5140000000000002e-06 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078511+0000", 2024-09-15T03:41:02.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1069e-05 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078516+0000", 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3310000000000003e-06 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080983+0000", 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024667840000000001 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.081067+0000", 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4568999999999996e-05 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.085340+0000", 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004272639 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.085442+0000", 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102571 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.087326+0000", 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018832549999999999 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:118 3.fs0 3:f17f8228:::smithi10536496-40:head [append 0~671744,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106135+0000", 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.50944540299999996, 2024-09-15T03:41:02.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063073440000000003, 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 118 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106135+0000", 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106135+0000", 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106137+0000", 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1650000000000001e-06 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106416+0000", 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027893599999999997 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106427+0000", 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1882000000000001e-05 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106432+0000", 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7550000000000002e-06 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106477+0000", 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4891000000000001e-05 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106607+0000", 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013030600000000001 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107378+0000", 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00077037999999999998 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.110631+0000", 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0032537180000000001 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112375+0000", 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017435949999999999 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112443+0000", 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7716e-05 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:119 3.fs0 3:f17f8228:::smithi10536496-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.50914111900000003, 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046131245000000001, 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.746 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 119 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2e-07 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106442+0000", 2024-09-15T03:41:02.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0150000000000002e-06 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106446+0000", 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6509999999999999e-06 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106448+0000", 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.21e-06 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107892+0000", 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001444435 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107983+0000", 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0179999999999994e-05 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108344+0000", 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036168399999999997 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.110942+0000", 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002597495 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152509+0000", 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041567518999999997 2024-09-15T03:41:02.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152571+0000", 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1636000000000004e-05 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:120 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.50911398299999999, 2024-09-15T03:41:02.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453923999999997, 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 120 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0400000000000002e-07 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106469+0000", 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.519e-06 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106474+0000", 2024-09-15T03:41:02.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5090000000000001e-06 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106480+0000", 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.285e-06 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108518+0000", 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020372820000000001 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108590+0000", 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2280999999999995e-05 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152611+0000", 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044021429000000001 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152655+0000", 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3671000000000003e-05 2024-09-15T03:41:02.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156921+0000", 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0042656439999999999 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:149 3.fs0 3:fc38c57e:::smithi10536496-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.150214+0000", 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.46536639699999999, 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045144404999999999, 2024-09-15T03:41:02.758 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 149 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150214+0000", 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150214+0000", 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150215+0000", 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1600000000000002e-07 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150217+0000", 2024-09-15T03:41:02.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6819999999999999e-06 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150223+0000", 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0240000000000003e-06 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150226+0000", 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8750000000000002e-06 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.153821+0000", 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035946250000000002 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.153869+0000", 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7809999999999998e-05 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156058+0000", 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0021892610000000001 2024-09-15T03:41:02.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156134+0000", 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5728000000000004e-05 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039224983999999997 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:175 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.42151753800000002, 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068787259000000003, 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 175 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:02.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194064+0000", 2024-09-15T03:41:02.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194065+0000", 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5540000000000001e-06 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194068+0000", 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0730000000000001e-06 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194071+0000", 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4770000000000002e-06 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195541+0000", 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014705670000000001 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195649+0000", 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108057 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:02.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.262851+0000", 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067201307000000002 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:02.768 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:02.769 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:02.770 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:02.770 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:02.770 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:02.771 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:02.771 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:02.771 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:02.772 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:02.772 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:02.772 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:02.773 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:02.773 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:02.774 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:02.774 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:02.768+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:02.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:02.776 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:02.797 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:02.797 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:02.814 INFO:tasks.rados.rados.0.smithi105.stdout:59: finishing write tid 3 to smithi10536496-13 2024-09-15T03:41:02.816 INFO:tasks.rados.rados.0.smithi105.stdout:59: finishing write tid 4 to smithi10536496-13 2024-09-15T03:41:02.816 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 9 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-15T03:41:02.816 INFO:tasks.rados.rados.0.smithi105.stdout:59: left oid 13 (ObjNum 53 snap 0 seq_num 53) 2024-09-15T03:41:02.816 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:02.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:02.835 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:02.856 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:02.856 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:02.870 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:02.869+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:02.876 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:02.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:02.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:02.875+0000 7fc44dcf1640 0 -- 172.21.15.5:0/3532874700 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7fc4481985a0 msgr2=0x7fc44819a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:02.910 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:02.910 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:02.915 INFO:tasks.rados.rados.0.smithi105.stdout:47: done (8 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:49: done (7 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:50: done (6 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:51: done (5 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:56: done (4 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:59: done (3 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:60: done (2 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:61: done (1 left) 2024-09-15T03:41:02.916 INFO:tasks.rados.rados.0.smithi105.stdout:62: delete oid 41 current snap is 0 2024-09-15T03:41:02.919 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:02.920 INFO:tasks.rados.rados.0.smithi105.stdout:62: done (1 left) 2024-09-15T03:41:02.920 INFO:tasks.rados.rados.0.smithi105.stdout:63: snap_create 2024-09-15T03:41:02.971 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:02.969+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:03.015 INFO:tasks.rados.rados.0.smithi105.stdout:63: done (1 left) 2024-09-15T03:41:03.015 INFO:tasks.rados.rados.0.smithi105.stdout:64: snap_create 2024-09-15T03:41:03.017 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:41:03.024 INFO:tasks.rados.rados.0.smithi105.stdout:53: finishing write tid 3 to smithi10536496-8 2024-09-15T03:41:03.025 INFO:tasks.rados.rados.0.smithi105.stdout:53: finishing write tid 4 to smithi10536496-8 2024-09-15T03:41:03.025 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 7 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-15T03:41:03.025 INFO:tasks.rados.rados.0.smithi105.stdout:53: left oid 8 (ObjNum 50 snap 0 seq_num 50) 2024-09-15T03:41:03.057 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:03.057 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:03.057 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:03.057 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:03.057 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:03.066 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:03.072 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:03.071+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:03.097 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:03.097 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:03.163 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.163 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.171 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:03.174 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:03.172+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:03.275 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.275 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.275 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:03.274+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:03.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:03.375 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:03.374+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:03.387 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.387 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:03.477 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:03.476+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:03.537 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.537 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.546 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:03.577 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:03.576+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:03.648 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.648 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:03.678 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:03.677+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:03.758 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.759 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:03.778 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:03.778+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:03.869 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.869 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:03.879 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:03.878+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:03.980 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:03.980 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:03.981 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:03.979+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:03.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:04.015 INFO:tasks.rados.rados.0.smithi105.stdout:53: done (1 left) 2024-09-15T03:41:04.015 INFO:tasks.rados.rados.0.smithi105.stdout:64: done (0 left) 2024-09-15T03:41:04.015 INFO:tasks.rados.rados.0.smithi105.stdout:65: delete oid 37 current snap is 2 2024-09-15T03:41:04.021 INFO:tasks.rados.rados.0.smithi105.stdout:65: done (0 left) 2024-09-15T03:41:04.021 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 3 current snap is 2 2024-09-15T03:41:04.021 INFO:tasks.rados.rados.0.smithi105.stdout:66: seq_num 57 ranges {49152=589824,638976=376832} 2024-09-15T03:41:04.025 INFO:tasks.rados.rados.0.smithi105.stdout:66: writing smithi10536496-3 from 49152 to 638976 tid 1 2024-09-15T03:41:04.027 INFO:tasks.rados.rados.0.smithi105.stdout:66: writing smithi10536496-3 from 638976 to 1015808 tid 2 2024-09-15T03:41:04.027 INFO:tasks.rados.rados.0.smithi105.stdout:67: setattr oid 10 current snap is 2 2024-09-15T03:41:04.028 INFO:tasks.rados.rados.0.smithi105.stdout:68: setattr oid 49 current snap is 2 2024-09-15T03:41:04.029 INFO:tasks.rados.rados.0.smithi105.stdout:69: read oid 41 snap -1 2024-09-15T03:41:04.029 INFO:tasks.rados.rados.0.smithi105.stdout:69: expect deleted 2024-09-15T03:41:04.029 INFO:tasks.rados.rados.0.smithi105.stdout:70: snap_remove snap 0 2024-09-15T03:41:04.081 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:04.080+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:04.094 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.095 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.103 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:04.181 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:04.180+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:04.205 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.205 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:04.297 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:04.296+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:04.355 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.356 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:04.383 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:04.382+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:04.485 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:04.483+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:04.507 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.507 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:04.585 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:04.585+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:04.657 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.657 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:04.710 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:04.709+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:04.780 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.780 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.786 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:04.786+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:04.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:04.887 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:04.886+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:04.932 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:04.932 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:04.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:04.987 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:04.986+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:05.022 INFO:tasks.rados.rados.0.smithi105.stdout:70: done (4 left) 2024-09-15T03:41:05.022 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 44 current snap is 2 2024-09-15T03:41:05.023 INFO:tasks.rados.rados.0.smithi105.stdout:71: seq_num 58 ranges {737280=606208,1343488=376832} 2024-09-15T03:41:05.029 INFO:tasks.rados.rados.0.smithi105.stdout:71: writing smithi10536496-44 from 737280 to 1343488 tid 1 2024-09-15T03:41:05.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:71: writing smithi10536496-44 from 1343488 to 1720320 tid 2 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:72: delete oid 48 current snap is 2 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:66: finishing write tid 1 to smithi10536496-3 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:66: finishing write tid 2 to smithi10536496-3 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 11 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:66: finishing write tid 3 to smithi10536496-3 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 13 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:66: finishing write tid 4 to smithi10536496-3 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 8 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-15T03:41:05.033 INFO:tasks.rados.rados.0.smithi105.stdout:66: left oid 3 (ObjNum 57 snap 2 seq_num 57) 2024-09-15T03:41:05.040 INFO:tasks.rados.rados.0.smithi105.stdout:71: finishing write tid 1 to smithi10536496-44 2024-09-15T03:41:05.040 INFO:tasks.rados.rados.0.smithi105.stdout:71: finishing write tid 2 to smithi10536496-44 2024-09-15T03:41:05.041 INFO:tasks.rados.rados.0.smithi105.stdout:66: done (5 left) 2024-09-15T03:41:05.042 INFO:tasks.rados.rados.0.smithi105.stdout:67: done (4 left) 2024-09-15T03:41:05.042 INFO:tasks.rados.rados.0.smithi105.stdout:68: done (3 left) 2024-09-15T03:41:05.042 INFO:tasks.rados.rados.0.smithi105.stdout:69: done (2 left) 2024-09-15T03:41:05.042 INFO:tasks.rados.rados.0.smithi105.stdout:72: done (1 left) 2024-09-15T03:41:05.042 INFO:tasks.rados.rados.0.smithi105.stdout:73: snap_remove snap 1 2024-09-15T03:41:05.050 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:05.050 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:05.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:05.089 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:05.088+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:05.109 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:05.190 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:05.189+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:05.200 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:05.201 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:05.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:05.278 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:05.278 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:05.291 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:05.290+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:05.298 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:05.298 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:05.310 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:05.310 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:05.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:05.392 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:05.392+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:05.420 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:05.420 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:05.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:05.493 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:05.492+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:05.520 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:05.530 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:05.530 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:05.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:05.542 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:05.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:05.560 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:05.580 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:05.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:05.601 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:05.622 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:05.622 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:05.625 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:05.624+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:05.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:05.641 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:05.662 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:05.662 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:05.682 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:05.682 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:05.683 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:05.683 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:05.683 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:05.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:05.694 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:05.694+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:05.702 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:05.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:05.722 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:05.722 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:05.740+0000 7fe6db7fe640 0 -- 172.21.15.5:0/4166715010 >> [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] conn(0x7fe6dc1900a0 msgr2=0x7fe6dc192480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:05.748 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:05.769 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:05.769 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:05.789 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:05.789 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:05.795 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:05.795+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:05.834 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8035275849999999, 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011155541, 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029084+0000", 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0603e-05 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029486+0000", 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000402144 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029493+0000", 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.165e-06 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029683+0000", 2024-09-15T03:41:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000190478 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029706+0000", 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2849000000000001e-05 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040193+0000", 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010486822999999999 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040228+0000", 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4993000000000002e-05 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:41, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8022886389999999, 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098373709999999993, 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030312+0000", 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030333+0000", 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0635000000000001e-05 2024-09-15T03:41:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030374+0000", 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030383+0000", 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000004e-06 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030436+0000", 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2605000000000002e-05 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030447+0000", 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1576999999999999e-05 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040120+0000", 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096721959999999992 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040149+0000", 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9218e-05 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8017039289999999, 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092836910000000002, 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030897+0000", 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6800000000000009e-07 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030911+0000", 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3589e-05 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030952+0000", 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1619000000000003e-05 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030962+0000", 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4679999999999998e-06 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031016+0000", 2024-09-15T03:41:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3671999999999997e-05 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031025+0000", 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6180000000000001e-06 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040164+0000", 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091392020000000008 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040180+0000", 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5554999999999999e-05 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.800977944, 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013955543000000001, 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031623+0000", 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031726+0000", 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103491 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031757+0000", 2024-09-15T03:41:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0908999999999998e-05 2024-09-15T03:41:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031770+0000", 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2565000000000001e-05 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031863+0000", 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2622999999999997e-05 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031928+0000", 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5984999999999999e-05 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045547+0000", 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013618655 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045578+0000", 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0565999999999998e-05 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.800014773, 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013036834000000001, 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032586+0000", 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032605+0000", 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-05 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032648+0000", 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2763999999999999e-05 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032656+0000", 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032693+0000", 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6701e-05 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032699+0000", 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6849999999999998e-06 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045592+0000", 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012893345 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045622+0000", 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0182e-05 2024-09-15T03:41:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7973773290000001, 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010405127, 2024-09-15T03:41:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035224+0000", 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1800000000000005e-07 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035236+0000", 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1952e-05 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035267+0000", 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035273+0000", 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035320+0000", 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6283999999999999e-05 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035326+0000", 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.105e-06 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045587+0000", 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010261562 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045628+0000", 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0469000000000002e-05 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7932433579999998, 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090014329999999997, 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039358+0000", 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039502+0000", 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000144272 2024-09-15T03:41:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039531+0000", 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8753999999999999e-05 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039537+0000", 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2009999999999996e-06 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039596+0000", 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9138000000000002e-05 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039605+0000", 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048329+0000", 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087234679999999998 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048358+0000", 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9442999999999999e-05 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=8, reqid=client.4562.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.040877+0000", 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7917232840000001, 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007439686, 2024-09-15T03:41:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040877+0000", 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040877+0000", 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040878+0000", 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2440000000000001e-06 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040889+0000", 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0621e-05 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040919+0000", 2024-09-15T03:41:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0358e-05 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040925+0000", 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0390000000000001e-06 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040972+0000", 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7076000000000001e-05 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040984+0000", 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1834000000000001e-05 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048277+0000", 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072927019999999999 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048317+0000", 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=7, reqid=client.4562.0:56, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.791029591, 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068524500000000004, 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999996e-07 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041600+0000", 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9142999999999999e-05 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041641+0000", 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0720999999999999e-05 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041647+0000", 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8479999999999997e-06 2024-09-15T03:41:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041686+0000", 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9289000000000003e-05 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041692+0000", 2024-09-15T03:41:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048373+0000", 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066808750000000002 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048423+0000", 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9818000000000003e-05 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=25, reqid=client.4562.0:101, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7482584239999999, 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011334317, 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084360+0000", 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7618999999999999e-05 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088093+0000", 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037333290000000001 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088098+0000", 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7010000000000004e-06 2024-09-15T03:41:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088131+0000", 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2653000000000001e-05 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088136+0000", 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4979999999999997e-06 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095632+0000", 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074962550000000003 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095676+0000", 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3516999999999997e-05 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=26, reqid=client.4562.0:103, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.087783+0000", 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7448167830000001, 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082797449999999998, 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087783+0000", 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087783+0000", 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087784+0000", 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8199999999999999e-07 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088078+0000", 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029388 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088869+0000", 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079139499999999997 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088882+0000", 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2745000000000001e-05 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088931+0000", 2024-09-15T03:41:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8918000000000002e-05 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088942+0000", 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096025+0000", 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070827989999999999 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096063+0000", 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8148999999999997e-05 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=27, reqid=client.4562.0:104, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7437637430000001, 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098078109999999996, 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088837+0000", 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088853+0000", 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5770999999999999e-05 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095603+0000", 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067504870000000003 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095612+0000", 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4139999999999994e-06 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095655+0000", 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3417000000000001e-05 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095665+0000", 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0006e-05 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098619+0000", 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029542119999999999 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098644+0000", 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4975999999999999e-05 2024-09-15T03:41:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=28, reqid=client.4562.0:109, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.095304+0000", 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.737295982, 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088261669999999993, 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095304+0000", 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095304+0000", 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095305+0000", 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.018e-06 2024-09-15T03:41:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095583+0000", 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027799500000000001 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096535+0000", 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095136699999999995 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096542+0000", 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.289e-06 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096602+0000", 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9731999999999997e-05 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096614+0000", 2024-09-15T03:41:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2115e-05 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104103+0000", 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074892919999999998 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104130+0000", 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7359e-05 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=29, reqid=client.4562.0:110, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.096493+0000", 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7361073290000002, 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076757680000000003, 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096493+0000", 2024-09-15T03:41:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096493+0000", 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096494+0000", 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4399999999999999e-07 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096512+0000", 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8726e-05 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099857+0000", 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033441619999999999 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099862+0000", 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5759999999999996e-06 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099907+0000", 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4409999999999997e-05 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099912+0000", 2024-09-15T03:41:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0390000000000004e-06 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104144+0000", 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042326580000000003 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104169+0000", 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4352999999999999e-05 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=30, to_read={3:eb414ad4:::smithi10536496-35:head=0,4096,0}, subchunks={3:eb414ad4:::smithi10536496-35:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7327684530000003, 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071743270000000003, 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099840+0000", 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6179999999999997e-06 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104946+0000", 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051057949999999998 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104951+0000", 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.729e-06 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104990+0000", 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9252999999999998e-05 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.107006+0000", 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020161990000000002 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=32, to_read={3:edf86ef6:::smithi10536496-37:head=0,4096,0}, subchunks={3:edf86ef6:::smithi10536496-37:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7271600579999999, 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039281828999999997, 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105441+0000", 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105447+0000", 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5010000000000001e-06 2024-09-15T03:41:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141752+0000", 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036305044000000002 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141762+0000", 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3940000000000002e-06 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141935+0000", 2024-09-15T03:41:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017359499999999999 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.144722+0000", 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00278671 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=34, reqid=client.4562.0:140, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.690087031, 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038895118999999999, 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142514+0000", 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.449e-05 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180934+0000", 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038406005999999999 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180939+0000", 2024-09-15T03:41:05.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4270000000000001e-06 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180984+0000", 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5751000000000003e-05 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180995+0000", 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0332e-05 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181383+0000", 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038812500000000002 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181408+0000", 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5349000000000001e-05 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=18, reqid=client.4562.0:143, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.145581+0000", 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6870189780000002, 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072727779999999997, 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145581+0000", 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145581+0000", 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145582+0000", 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145785+0000", 2024-09-15T03:41:05.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020355199999999999 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145809+0000", 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3300000000000001e-05 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145815+0000", 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.494e-06 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145861+0000", 2024-09-15T03:41:05.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6054999999999999e-05 2024-09-15T03:41:05.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145879+0000", 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8281999999999999e-05 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.152829+0000", 2024-09-15T03:41:05.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069491350000000004 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.152854+0000", 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5304999999999999e-05 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=39, reqid=client.4562.0:163, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:05.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6475992189999999, 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099943480000000001, 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185002+0000", 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185014+0000", 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.205e-05 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194530+0000", 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095162920000000008 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194535+0000", 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4090000000000002e-06 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194590+0000", 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4234000000000003e-05 2024-09-15T03:41:05.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194595+0000", 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4369999999999998e-06 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194977+0000", 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038248600000000002 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194995+0000", 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799e-05 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=40, reqid=client.4562.0:176, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6380997390000003, 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229890967, 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:05.928 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:05.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194501+0000", 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194514+0000", 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423889+0000", 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229374775 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423894+0000", 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0669999999999999e-06 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423938+0000", 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3967000000000001e-05 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423948+0000", 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2920000000000001e-06 2024-09-15T03:41:05.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424370+0000", 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042234599999999998 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424391+0000", 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1441999999999999e-05 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:05.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:05.933 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:05.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:05.935 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:05.933+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:06.026 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:73: done (1 left) 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:74: copy_from oid 7 from oid 45 current snap is 2 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:71: finishing write tid 3 to smithi10536496-44 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:71: finishing write tid 4 to smithi10536496-44 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 13 (ObjNum 58 snap 2 seq_num 58) dirty exists 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:71: left oid 44 (ObjNum 58 snap 2 seq_num 58) 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:71: done (1 left) 2024-09-15T03:41:06.030 INFO:tasks.rados.rados.0.smithi105.stdout:75: rollback oid 50 current snap is 2 2024-09-15T03:41:06.031 INFO:tasks.rados.rados.0.smithi105.stdout:75: done (1 left) 2024-09-15T03:41:06.031 INFO:tasks.rados.rados.0.smithi105.stdout:76: snap_create 2024-09-15T03:41:06.035 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:06.034+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:06.041 INFO:tasks.rados.rados.0.smithi105.stdout:74: finishing copy_from to smithi10536496-7 2024-09-15T03:41:06.041 INFO:tasks.rados.rados.0.smithi105.stdout:74: got expected ENOENT (src dne) 2024-09-15T03:41:06.041 INFO:tasks.rados.rados.0.smithi105.stdout:74: finishing copy_from racing read to smithi10536496-7 2024-09-15T03:41:06.081 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:06.081 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:06.081 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:06.081 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:06.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:06.136 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:06.135+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:06.191 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:06.192 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:06.192 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:06.192 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:06.192 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:06.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:06.237 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:06.236+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:06.303 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:06.303 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:06.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:06.303 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:06.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.303 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451897940999999, 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013249957, 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 50 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849363+0000", 2024-09-15T03:41:06.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2099999999999997e-07 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849419+0000", 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6660999999999998e-05 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849432+0000", 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2313999999999999e-05 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849437+0000", 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6130000000000003e-06 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854814+0000", 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053769459999999996 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854864+0000", 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9937e-05 2024-09-15T03:41:06.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854960+0000", 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6176999999999998e-05 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860859+0000", 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058984600000000003 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862557+0000", 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016980459999999999 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862590+0000", 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3015999999999997e-05 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862612+0000", 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2266e-05 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451806548, 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.310 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:41:06.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451778442999998, 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:06.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:41:06.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:41:06.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:55 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.437982331000001, 2024-09-15T03:41:06.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011236095999999999, 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 55 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863278+0000", 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863353+0000", 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4338999999999994e-05 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863359+0000", 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000003e-06 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863364+0000", 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3649999999999997e-06 2024-09-15T03:41:06.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867028+0000", 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036646890000000001 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867081+0000", 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2984e-05 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867157+0000", 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6079000000000001e-05 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.870092+0000", 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029341810000000001 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.874426+0000", 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043345409999999999 2024-09-15T03:41:06.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.874474+0000", 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8285000000000003e-05 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.874514+0000", 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9508999999999997e-05 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:56 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.437891661999998, 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012488551000000001, 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 56 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.329 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863369+0000", 2024-09-15T03:41:06.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863435+0000", 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6039999999999998e-05 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863441+0000", 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8250000000000001e-06 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863444+0000", 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2739999999999999e-06 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867710+0000", 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042658740000000002 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867818+0000", 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010801100000000001 2024-09-15T03:41:06.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867963+0000", 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014490799999999999 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.872812+0000", 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048489290000000001 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875774+0000", 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029621589999999998 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875821+0000", 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6966999999999999e-05 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875857+0000", 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6059999999999997e-05 2024-09-15T03:41:06.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.157010328999998, 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012203834, 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.335 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 97 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1699999999999998e-07 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144254+0000", 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8039999999999999e-06 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144259+0000", 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3310000000000003e-06 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144262+0000", 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490000000000002e-06 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151586+0000", 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0073231959999999997 2024-09-15T03:41:06.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151660+0000", 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4589e-05 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151733+0000", 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2595000000000005e-05 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154896+0000", 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031635869999999998 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156408+0000", 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001511996 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156435+0000", 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6917999999999999e-05 2024-09-15T03:41:06.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156454+0000", 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8451999999999999e-05 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.156995007999999, 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012289876999999999, 2024-09-15T03:41:06.341 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 98 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.34e-07 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144267+0000", 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.795e-06 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144270+0000", 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1760000000000002e-06 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144273+0000", 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:06.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152322+0000", 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080492889999999994 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152365+0000", 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3634999999999998e-05 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152434+0000", 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8738999999999994e-05 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154902+0000", 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024674879999999999 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156492+0000", 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015906309999999999 2024-09-15T03:41:06.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156520+0000", 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.811e-05 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156555+0000", 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4585000000000002e-05 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.156985195000001, 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013168571, 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.348 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 99 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144313+0000", 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7694000000000002e-05 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144317+0000", 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2719999999999999e-06 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144320+0000", 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152651+0000", 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083311770000000004 2024-09-15T03:41:06.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152696+0000", 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4750999999999999e-05 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152767+0000", 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0850000000000001e-05 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156577+0000", 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038109319999999999 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157385+0000", 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080757500000000005 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157410+0000", 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5306000000000001e-05 2024-09-15T03:41:06.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157444+0000", 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3192999999999999e-05 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:06.354 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.156937336999999, 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:41:06.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:41:06.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:41:06.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:12 3.3s0 3:cce4e8f8:::smithi10536496-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018097+0000", 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2831628649999995, 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012025098, 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018097+0000", 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018097+0000", 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018098+0000", 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018102+0000", 2024-09-15T03:41:06.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6940000000000001e-06 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018109+0000", 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4749999999999996e-06 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018123+0000", 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4192000000000001e-05 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021889+0000", 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037652800000000002 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5224999999999997e-05 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027411+0000", 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054271440000000001 2024-09-15T03:41:06.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027497+0000", 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6318999999999998e-05 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030122+0000", 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026250959999999999 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=8, to_read={3:fff44eab:::smithi10536496-10:head=0,4096,0}, subchunks={3:fff44eab:::smithi10536496-10:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2673609929999996, 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046423195, 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033900+0000", 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033906+0000", 2024-09-15T03:41:06.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080075+0000", 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046169038000000003 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080080+0000", 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080125+0000", 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5151000000000002e-05 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080322+0000", 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019686500000000001 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:96 3.2s0 3:41f9c41d:::smithi10536496-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2264934079999996, 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013049477, 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 96 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:41:06.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074770+0000", 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8890000000000002e-06 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074776+0000", 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074779+0000", 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3759999999999999e-06 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.077403+0000", 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026236649999999999 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.077475+0000", 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2298999999999993e-05 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.085774+0000", 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008299117 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.085861+0000", 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6575000000000004e-05 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087816+0000", 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019556719999999999 2024-09-15T03:41:06.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 35/32 ECSubWriteReply(tid=118, last_complete=35'2, committed=1, applied=1))", 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.214428711, 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021508349, 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086832+0000", 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:41:06.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086835+0000", 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0479999999999999e-06 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108275+0000", 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021439848000000001 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108283+0000", 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8609999999999998e-06 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108320+0000", 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7330999999999999e-05 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108340+0000", 2024-09-15T03:41:06.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9531999999999999e-05 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1928553449999999, 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041863826999999999, 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.382 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108419+0000", 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149106+0000", 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040686870999999999 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149110+0000", 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.335e-06 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149157+0000", 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7131e-05 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149164+0000", 2024-09-15T03:41:06.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150245+0000", 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010804580000000001 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150269+0000", 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4128e-05 2024-09-15T03:41:06.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:129 3.3s0 3:c25886c1:::smithi10536496-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1811044470000001, 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012678227, 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0100000000000005e-07 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120160+0000", 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.337e-06 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120167+0000", 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120171+0000", 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4179999999999999e-06 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.122623+0000", 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002451646 2024-09-15T03:41:06.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.122711+0000", 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999999e-05 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.127891+0000", 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051798429999999999 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.127977+0000", 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5449000000000001e-05 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.132834+0000", 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048573330000000001 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:132 3.3s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.392 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1760459680000004, 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015465478, 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 132 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125215+0000", 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125216+0000", 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7349999999999999e-06 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125223+0000", 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125227+0000", 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8840000000000004e-06 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129903+0000", 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046767349999999996 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129995+0000", 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1706000000000006e-05 2024-09-15T03:41:06.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133187+0000", 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031916510000000002 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133234+0000", 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7862999999999999e-05 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140680+0000", 2024-09-15T03:41:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074451860000000003 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:133 3.3s0 3:ca2bd2fa:::smithi10536496-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.129629+0000", 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1716307050000001, 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011270361, 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129629+0000", 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129629+0000", 2024-09-15T03:41:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129630+0000", 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4099999999999997e-07 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130352+0000", 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072137799999999995 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130361+0000", 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130365+0000", 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9650000000000002e-06 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130443+0000", 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8471000000000001e-05 2024-09-15T03:41:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130576+0000", 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013331199999999999 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.131404+0000", 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00082763200000000002 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.132975+0000", 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015708619999999999 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140821+0000", 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078463049999999996 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140900+0000", 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8632000000000002e-05 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:135 3.3s0 3:ca2bd2fa:::smithi10536496-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.130397+0000", 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1708627470000001, 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01259819, 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 135 2024-09-15T03:41:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130397+0000", 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130397+0000", 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130398+0000", 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7800000000000002e-07 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130399+0000", 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130404+0000", 2024-09-15T03:41:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.318e-06 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130406+0000", 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.481e-06 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133695+0000", 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032885599999999998 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133744+0000", 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9679999999999999e-05 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141649+0000", 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079043429999999994 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141725+0000", 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5959999999999998e-05 2024-09-15T03:41:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142996+0000", 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001270923 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=25, to_read={3:fc38c57e:::smithi10536496-49:head=0,4096,0}, subchunks={3:fc38c57e:::smithi10536496-49:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.144805206, 2024-09-15T03:41:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038157982, 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156460+0000", 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194316+0000", 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037855609999999998 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194320+0000", 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999998e-06 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194361+0000", 2024-09-15T03:41:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0708999999999999e-05 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194613+0000", 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025227700000000002 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=31, to_read={3:f17f8228:::smithi10536496-40:head=0,167936,0}, subchunks={3:f17f8228:::smithi10536496-40:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1053570160000001, 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066290004, 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195904+0000", 2024-09-15T03:41:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195911+0000", 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3030000000000002e-06 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262021+0000", 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066109629000000003 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262025+0000", 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5909999999999998e-06 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262085+0000", 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0183999999999998e-05 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262193+0000", 2024-09-15T03:41:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107648 2024-09-15T03:41:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:06.418 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:06.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:06.420 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:06.418+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:06.520 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:06.519+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:06.530 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:06.531 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:06.540 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:06.621 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:06.619+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:06.679 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:06.679 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:06.687 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:06.721 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:06.721+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:06.726 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T03:41:06.726 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:41:06.726 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-15T03:41:06.726 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:41:06.786 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:50 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'32, mlcod=24'23)", 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:43.855076+0000", 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.929719584000001, 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073421220000000004, 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855076+0000", 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855076+0000", 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855077+0000", 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855126+0000", 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9027999999999999e-05 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855154+0000", 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8011999999999999e-05 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.855159+0000", 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3590000000000001e-06 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856555+0000", 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001396226 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.856574+0000", 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9476000000000001e-05 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862388+0000", 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005813688 2024-09-15T03:41:06.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:43.862419+0000", 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0604e-05 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:66 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'44, mlcod=24'40)", 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:06.792 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.772255764000001, 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077416819999999997, 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012541+0000", 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2e-07 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012592+0000", 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1273999999999997e-05 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012612+0000", 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0389000000000001e-05 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012615+0000", 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1250000000000001e-06 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012636+0000", 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1070999999999999e-05 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012647+0000", 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.08e-05 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020245+0000", 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075975579999999999 2024-09-15T03:41:06.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020282+0000", 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7045000000000002e-05 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:67 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'45, mlcod=24'40)", 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:06.797 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.771929263000001, 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074323139999999998, 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:06.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012867+0000", 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012870+0000", 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2610000000000002e-06 2024-09-15T03:41:06.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012888+0000", 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8046e-05 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012891+0000", 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0170000000000001e-06 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012916+0000", 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4307e-05 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012928+0000", 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1956e-05 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020284+0000", 2024-09-15T03:41:06.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073567820000000001 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020299+0000", 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4555e-05 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.802 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:68 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'46, mlcod=24'40)", 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.771523383999998, 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070388200000000003, 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013273+0000", 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013276+0000", 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013293+0000", 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6504999999999999e-05 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013295+0000", 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7599999999999998e-06 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013319+0000", 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3272999999999999e-05 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.013330+0000", 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1759999999999999e-05 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020300+0000", 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069692130000000001 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020311+0000", 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.173e-05 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:100 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'74, mlcod=24'65)", 2024-09-15T03:41:06.807 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.154603+0000", 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.630192658999999, 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072170230000000004, 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154603+0000", 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154603+0000", 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154604+0000", 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6700000000000003e-07 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154625+0000", 2024-09-15T03:41:06.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1237e-05 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154648+0000", 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2807999999999999e-05 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154652+0000", 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.07e-06 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154675+0000", 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3394e-05 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.154687+0000", 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1858999999999999e-05 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.161792+0000", 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071049260000000001 2024-09-15T03:41:06.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.161820+0000", 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8161999999999999e-05 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:4 3.es0 3:726216d4:::smithi10536496-2:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016531+0000", 2024-09-15T03:41:06.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7682652040000004, 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073425390000000004, 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 4 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016531+0000", 2024-09-15T03:41:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016531+0000", 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016532+0000", 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.655e-06 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016603+0000", 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0857000000000003e-05 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016614+0000", 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0324e-05 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016620+0000", 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4849999999999997e-06 2024-09-15T03:41:06.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016673+0000", 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3257999999999997e-05 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016777+0000", 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010320899999999999 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017147+0000", 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037069700000000002 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019205+0000", 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020578580000000001 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023801+0000", 2024-09-15T03:41:06.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045960560000000003 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023873+0000", 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2139999999999997e-05 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:5 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:06.818 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7681628519999997, 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010334054000000001, 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016634+0000", 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7000000000000004e-07 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016637+0000", 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6280000000000001e-06 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8289999999999997e-06 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016649+0000", 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017615+0000", 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096610800000000003 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017698+0000", 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3428000000000002e-05 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018129+0000", 2024-09-15T03:41:06.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043087300000000002 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022895+0000", 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047655759999999997 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026926+0000", 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040308160000000004 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026967+0000", 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1554999999999998e-05 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:6 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7681309499999998, 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012850047, 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6100000000000001e-07 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016667+0000", 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9769999999999999e-06 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016674+0000", 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016678+0000", 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000003e-06 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018325+0000", 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001646385 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6032999999999995e-05 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027234+0000", 2024-09-15T03:41:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088331369999999996 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027316+0000", 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1862999999999999e-05 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029515+0000", 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002199281 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/32 ECSubWrite(tid=2, reqid=client.4562.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024402+0000", 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7603939909999999, 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070959609999999996, 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024402+0000", 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024402+0000", 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024402+0000", 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0100000000000001e-07 2024-09-15T03:41:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024516+0000", 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114267 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024545+0000", 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8501e-05 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024551+0000", 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1480000000000003e-06 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024598+0000", 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6588000000000002e-05 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024605+0000", 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7230000000000004e-06 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031475+0000", 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068694769999999997 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031498+0000", 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3056000000000001e-05 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s5 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.74553768, 2024-09-15T03:41:06.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079303010000000007, 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1000000000000001e-07 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039586+0000", 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032722799999999999 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039615+0000", 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9383e-05 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039621+0000", 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1029999999999998e-06 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039673+0000", 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2261e-05 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039682+0000", 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7649999999999999e-06 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047166+0000", 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074841710000000004 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047189+0000", 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.228e-05 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.841 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:58 3.es0 3:7dd49206:::smithi10536496-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7445071609999996, 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083733009999999997, 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1990000000000001e-06 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040639+0000", 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034862499999999998 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040646+0000", 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6290000000000001e-06 2024-09-15T03:41:06.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040650+0000", 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.552e-06 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040691+0000", 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.07e-05 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040781+0000", 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0859000000000005e-05 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041560+0000", 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00077845500000000005 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044707+0000", 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031467969999999998 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048626+0000", 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039196150000000004 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048662+0000", 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.587e-05 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:59 3.es0 3:7dd49206:::smithi10536496-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.744139648, 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084161029999999994, 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 59 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040657+0000", 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4899999999999995e-07 2024-09-15T03:41:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040659+0000", 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3319999999999999e-06 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040665+0000", 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6690000000000003e-06 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040668+0000", 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4630000000000001e-06 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041990+0000", 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013215220000000001 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042074+0000", 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4060000000000005e-05 2024-09-15T03:41:06.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042441+0000", 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000367432 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047193+0000", 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047516440000000002 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049024+0000", 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001831378 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049072+0000", 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8053999999999998e-05 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:60 3.es0 3:7dd49206:::smithi10536496-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.853 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.744123858, 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010867740000000001, 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 60 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040673+0000", 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040675+0000", 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2479999999999999e-06 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040681+0000", 2024-09-15T03:41:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0170000000000002e-06 2024-09-15T03:41:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040689+0000", 2024-09-15T03:41:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6330000000000002e-06 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042594+0000", 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019041489999999999 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042676+0000", 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2633999999999995e-05 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049349+0000", 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066727380000000001 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049419+0000", 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9868000000000003e-05 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.051540+0000", 2024-09-15T03:41:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002121029 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:83 3.es0 3:7f98869d:::smithi10536496-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7221529389999999, 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080589019999999997, 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 83 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062645+0000", 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9199999999999998e-06 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062651+0000", 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062654+0000", 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6729999999999998e-06 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.063750+0000", 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001095525 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.063801+0000", 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1573000000000002e-05 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064048+0000", 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024709100000000001 2024-09-15T03:41:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064653+0000", 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060490800000000003 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070635+0000", 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059821070000000004 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070702+0000", 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6548e-05 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:84 3.es0 3:7f98869d:::smithi10536496-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7221187249999996, 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011714999, 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 84 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062678+0000", 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062679+0000", 2024-09-15T03:41:06.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6920000000000001e-06 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062685+0000", 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0979999999999999e-06 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062688+0000", 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3860000000000001e-06 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064229+0000", 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015411999999999999 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064308+0000", 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8621000000000005e-05 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070977+0000", 2024-09-15T03:41:06.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006669538 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.071049+0000", 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1464000000000006e-05 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.074392+0000", 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003343586 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.3s4 35/32 ECSubWrite(tid=126, reqid=client.4562.0:134, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.6523451339999999, 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078231129999999996, 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:06.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.44e-07 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132468+0000", 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6424999999999999e-05 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132496+0000", 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7798e-05 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132500+0000", 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2930000000000002e-06 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132531+0000", 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1028999999999997e-05 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132537+0000", 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.699e-06 2024-09-15T03:41:06.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.140252+0000", 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007715431 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.140274+0000", 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1793999999999998e-05 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:152 3.es0 3:71615b7a:::smithi10536496-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.6291836850000001, 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.031528529, 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 152 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155613+0000", 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155616+0000", 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.88e-06 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155623+0000", 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6950000000000003e-06 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155627+0000", 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999997e-06 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156746+0000", 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001119229 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156828+0000", 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2294999999999997e-05 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157113+0000", 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028439999999999997 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.159322+0000", 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022090719999999999 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187068+0000", 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027746694999999998 2024-09-15T03:41:06.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187141+0000", 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:153 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:06.883 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.6291623350000002, 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.032862542000000002, 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 153 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:06.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155636+0000", 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7260000000000001e-06 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155642+0000", 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1449999999999996e-06 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155645+0000", 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4470000000000002e-06 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157320+0000", 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016747769999999999 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157366+0000", 2024-09-15T03:41:06.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5982999999999997e-05 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187194+0000", 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.029827955 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187240+0000", 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5494000000000002e-05 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188496+0000", 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012565250000000001 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:166 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5992768440000003, 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.10104360799999999, 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 166 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185520+0000", 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4099999999999999e-07 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185522+0000", 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.588e-06 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185526+0000", 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5130000000000001e-06 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185528+0000", 2024-09-15T03:41:06.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3099999999999999e-06 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186513+0000", 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098462300000000001 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8233000000000003e-05 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186810+0000", 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023878899999999999 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187938+0000", 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001128698 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286460+0000", 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098521759 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286563+0000", 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102654 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:167 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.599257572, 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.101989948, 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 167 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185539+0000", 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8599999999999999e-07 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185540+0000", 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4020000000000001e-06 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185544+0000", 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6619999999999998e-06 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185547+0000", 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6330000000000002e-06 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188004+0000", 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024565619999999998 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188048+0000", 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3925999999999999e-05 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286831+0000", 2024-09-15T03:41:06.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098782707999999997 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286925+0000", 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4609000000000002e-05 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.287528+0000", 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060316000000000005 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:06.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:06.902 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:06.903 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:06.901+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:06.904 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:07.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:07.002 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:07.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.003 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:07.003+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:07.011 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:07.018 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:41:07.019 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:41:07.028 INFO:tasks.rados.rados.0.smithi105.stdout:74: done (1 left) 2024-09-15T03:41:07.028 INFO:tasks.rados.rados.0.smithi105.stdout:76: done (0 left) 2024-09-15T03:41:07.028 INFO:tasks.rados.rados.0.smithi105.stdout:77: snap_remove snap 2 2024-09-15T03:41:07.030 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-15T03:41:06.024229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17990195,"num_objects":40,"num_object_clones":1,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":165,"num_read_kb":112,"num_write":255,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":4777060,"kb_used_data":19336,"kb_used_omap":26,"kb_used_meta":4757477,"kb_avail":1505172380,"statfs":{"total":1546188226560,"available":1541296517120,"internally_reserved":0,"allocated":19800064,"data_stored":16538192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27264,"internal_metadata":4871656832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17399808,"num_objects":36,"num_object_clones":1,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":79,"num_read_kb":39,"num_write":108,"num_write_kb":18096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008153"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.588519+0000","last_change":"2024-09-15T03:40:59.585492+0000","last_active":"2024-09-15T03:40:59.588519+0000","last_peered":"2024-09-15T03:40:59.588519+0000","last_clean":"2024-09-15T03:40:59.588519+0000","last_became_active":"2024-09-15T03:40:59.584644+0000","last_became_peered":"2024-09-15T03:40:59.584644+0000","last_unstale":"2024-09-15T03:40:59.588519+0000","last_undegraded":"2024-09-15T03:40:59.588519+0000","last_fullsized":"2024-09-15T03:40:59.588519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:07.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"35'12","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.286462+0000","last_change":"2024-09-15T03:40:59.757338+0000","last_active":"2024-09-15T03:41:02.286462+0000","last_peered":"2024-09-15T03:41:02.286462+0000","last_clean":"2024-09-15T03:41:02.286462+0000","last_became_active":"2024-09-15T03:40:59.756671+0000","last_became_peered":"2024-09-15T03:40:59.756671+0000","last_unstale":"2024-09-15T03:41:02.286462+0000","last_undegraded":"2024-09-15T03:41:02.286462+0000","last_fullsized":"2024-09-15T03:41:02.286462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.425614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,1,11,8,6,12],"acting":[2,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.585986+0000","last_change":"2024-09-15T03:40:59.585080+0000","last_active":"2024-09-15T03:40:59.585986+0000","last_peered":"2024-09-15T03:40:59.585986+0000","last_clean":"2024-09-15T03:40:59.585986+0000","last_became_active":"2024-09-15T03:40:59.584469+0000","last_became_peered":"2024-09-15T03:40:59.584469+0000","last_unstale":"2024-09-15T03:40:59.585986+0000","last_undegraded":"2024-09-15T03:40:59.585986+0000","last_fullsized":"2024-09-15T03:40:59.585986+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:17.875841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.588513+0000","last_change":"2024-09-15T03:40:59.585372+0000","last_active":"2024-09-15T03:40:59.588513+0000","last_peered":"2024-09-15T03:40:59.588513+0000","last_clean":"2024-09-15T03:40:59.588513+0000","last_became_active":"2024-09-15T03:40:59.584403+0000","last_became_peered":"2024-09-15T03:40:59.584403+0000","last_unstale":"2024-09-15T03:40:59.588513+0000","last_undegraded":"2024-09-15T03:40:59.588513+0000","last_fullsized":"2024-09-15T03:40:59.588513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:21.575002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"35'7","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.182892+0000","last_change":"2024-09-15T03:40:59.757127+0000","last_active":"2024-09-15T03:41:02.182892+0000","last_peered":"2024-09-15T03:41:02.182892+0000","last_clean":"2024-09-15T03:41:02.182892+0000","last_became_active":"2024-09-15T03:40:59.756158+0000","last_became_peered":"2024-09-15T03:40:59.756158+0000","last_unstale":"2024-09-15T03:41:02.182892+0000","last_undegraded":"2024-09-15T03:41:02.182892+0000","last_fullsized":"2024-09-15T03:41:02.182892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:18.180847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1343488,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":1424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"35'12","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.166513+0000","last_change":"2024-09-15T03:40:59.756215+0000","last_active":"2024-09-15T03:41:02.166513+0000","last_peered":"2024-09-15T03:41:02.166513+0000","last_clean":"2024-09-15T03:41:02.166513+0000","last_became_active":"2024-09-15T03:40:59.755360+0000","last_became_peered":"2024-09-15T03:40:59.755360+0000","last_unstale":"2024-09-15T03:41:02.166513+0000","last_undegraded":"2024-09-15T03:41:02.166513+0000","last_fullsized":"2024-09-15T03:41:02.166513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:21.475353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2785280,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":12,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"35'7","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.169733+0000","last_change":"2024-09-15T03:40:59.585686+0000","last_active":"2024-09-15T03:41:02.169733+0000","last_peered":"2024-09-15T03:41:02.169733+0000","last_clean":"2024-09-15T03:41:02.169733+0000","last_became_active":"2024-09-15T03:40:59.585033+0000","last_became_peered":"2024-09-15T03:40:59.585033+0000","last_unstale":"2024-09-15T03:41:02.169733+0000","last_undegraded":"2024-09-15T03:41:02.169733+0000","last_fullsized":"2024-09-15T03:41:02.169733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:19.003756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1228800,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.8","version":"35'8","reported_seq":27,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.140859+0000","last_change":"2024-09-15T03:40:59.690870+0000","last_active":"2024-09-15T03:41:02.140859+0000","last_peered":"2024-09-15T03:41:02.140859+0000","last_clean":"2024-09-15T03:41:02.140859+0000","last_became_active":"2024-09-15T03:40:59.690393+0000","last_became_peered":"2024-09-15T03:40:59.690393+0000","last_unstale":"2024-09-15T03:41:02.140859+0000","last_undegraded":"2024-09-15T03:41:02.140859+0000","last_fullsized":"2024-09-15T03:41:02.140859+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:07.413469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1736704,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":8,"num_write_kb":1696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":154,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:41:00.040472+0000","last_change":"2024-09-15T03:40:43.639536+0000","last_active":"2024-09-15T03:41:00.040472+0000","last_peered":"2024-09-15T03:41:00.040472+0000","last_clean":"2024-09-15T03:41:00.040472+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:41:00.040472+0000","last_undegraded":"2024-09-15T03:41:00.040472+0000","last_fullsized":"2024-09-15T03:41:00.040472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:40:42.622204+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:49.505752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":21,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:41:00.128314+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:41:00.128314+0000","last_peered":"2024-09-15T03:41:00.128314+0000","last_clean":"2024-09-15T03:41:00.128314+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:41:00.128314+0000","last_undegraded":"2024-09-15T03:41:00.128314+0000","last_fullsized":"2024-09-15T03:41:00.128314+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"35'2","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.085479+0000","last_change":"2024-09-15T03:40:59.125745+0000","last_active":"2024-09-15T03:41:02.085479+0000","last_peered":"2024-09-15T03:41:02.085479+0000","last_clean":"2024-09-15T03:41:02.085479+0000","last_became_active":"2024-09-15T03:40:59.125069+0000","last_became_peered":"2024-09-15T03:40:59.125069+0000","last_unstale":"2024-09-15T03:41:02.085479+0000","last_undegraded":"2024-09-15T03:41:02.085479+0000","last_fullsized":"2024-09-15T03:41:02.085479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.735835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,3,6,15,1,11],"acting":[13,2,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:41:00.128588+0000","last_change":"2024-09-15T03:40:48.666791+0000","last_active":"2024-09-15T03:41:00.128588+0000","last_peered":"2024-09-15T03:41:00.128588+0000","last_clean":"2024-09-15T03:41:00.128588+0000","last_became_active":"2024-09-15T03:40:46.657361+0000","last_became_peered":"2024-09-15T03:40:46.657361+0000","last_unstale":"2024-09-15T03:41:00.128588+0000","last_undegraded":"2024-09-15T03:41:00.128588+0000","last_fullsized":"2024-09-15T03:41:00.128588+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:06.411518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000334655,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"35'6","reported_seq":24,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.077204+0000","last_change":"2024-09-15T03:40:59.756099+0000","last_active":"2024-09-15T03:41:02.077204+0000","last_peered":"2024-09-15T03:41:02.077204+0000","last_clean":"2024-09-15T03:41:02.077204+0000","last_became_active":"2024-09-15T03:40:59.755521+0000","last_became_peered":"2024-09-15T03:40:59.755521+0000","last_unstale":"2024-09-15T03:41:02.077204+0000","last_undegraded":"2024-09-15T03:41:02.077204+0000","last_fullsized":"2024-09-15T03:41:02.077204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:25.610966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196032,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,2,13],"acting":[4,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:41:00.040336+0000","last_change":"2024-09-15T03:40:50.313452+0000","last_active":"2024-09-15T03:41:00.040336+0000","last_peered":"2024-09-15T03:41:00.040336+0000","last_clean":"2024-09-15T03:41:00.040336+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:41:00.040336+0000","last_undegraded":"2024-09-15T03:41:00.040336+0000","last_fullsized":"2024-09-15T03:41:00.040336+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.213415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"35'10","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.163429+0000","last_change":"2024-09-15T03:40:59.126492+0000","last_active":"2024-09-15T03:41:02.163429+0000","last_peered":"2024-09-15T03:41:02.163429+0000","last_clean":"2024-09-15T03:41:02.163429+0000","last_became_active":"2024-09-15T03:40:59.125536+0000","last_became_peered":"2024-09-15T03:40:59.125536+0000","last_unstale":"2024-09-15T03:41:02.163429+0000","last_undegraded":"2024-09-15T03:41:02.163429+0000","last_fullsized":"2024-09-15T03:41:02.163429+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:21.365264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1261568,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":10,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,4],"acting":[14,2,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":23,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:41:00.128742+0000","last_change":"2024-09-15T03:40:48.667305+0000","last_active":"2024-09-15T03:41:00.128742+0000","last_peered":"2024-09-15T03:41:00.128742+0000","last_clean":"2024-09-15T03:41:00.128742+0000","last_became_active":"2024-09-15T03:40:46.775427+0000","last_became_peered":"2024-09-15T03:40:46.775427+0000","last_unstale":"2024-09-15T03:41:00.128742+0000","last_undegraded":"2024-09-15T03:41:00.128742+0000","last_fullsized":"2024-09-15T03:41:00.128742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:00.077013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084937499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"35'9","reported_seq":28,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.142968+0000","last_change":"2024-09-15T03:40:59.131823+0000","last_active":"2024-09-15T03:41:02.142968+0000","last_peered":"2024-09-15T03:41:02.142968+0000","last_clean":"2024-09-15T03:41:02.142968+0000","last_became_active":"2024-09-15T03:40:59.131140+0000","last_became_peered":"2024-09-15T03:40:59.131140+0000","last_unstale":"2024-09-15T03:41:02.142968+0000","last_undegraded":"2024-09-15T03:41:02.142968+0000","last_fullsized":"2024-09-15T03:41:02.142968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:06.578837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1523712,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":9,"num_write_kb":2224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,2,5,3],"acting":[13,7,4,15,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.125579+0000","last_change":"2024-09-15T03:40:59.124775+0000","last_active":"2024-09-15T03:40:59.125579+0000","last_peered":"2024-09-15T03:40:59.125579+0000","last_clean":"2024-09-15T03:40:59.125579+0000","last_became_active":"2024-09-15T03:40:59.124035+0000","last_became_peered":"2024-09-15T03:40:59.124035+0000","last_unstale":"2024-09-15T03:40:59.125579+0000","last_undegraded":"2024-09-15T03:40:59.125579+0000","last_fullsized":"2024-09-15T03:40:59.125579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:07.553804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:41:00.040059+0000","last_change":"2024-09-15T03:40:49.005549+0000","last_active":"2024-09-15T03:41:00.040059+0000","last_peered":"2024-09-15T03:41:00.040059+0000","last_clean":"2024-09-15T03:41:00.040059+0000","last_became_active":"2024-09-15T03:40:46.669330+0000","last_became_peered":"2024-09-15T03:40:46.669330+0000","last_unstale":"2024-09-15T03:41:00.040059+0000","last_undegraded":"2024-09-15T03:41:00.040059+0000","last_fullsized":"2024-09-15T03:41:00.040059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:12.749616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000296755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.997272+0000","last_change":"2024-09-15T03:40:59.124384+0000","last_active":"2024-09-15T03:40:59.997272+0000","last_peered":"2024-09-15T03:40:59.997272+0000","last_clean":"2024-09-15T03:40:59.997272+0000","last_became_active":"2024-09-15T03:40:59.123847+0000","last_became_peered":"2024-09-15T03:40:59.123847+0000","last_unstale":"2024-09-15T03:40:59.997272+0000","last_undegraded":"2024-09-15T03:40:59.997272+0000","last_fullsized":"2024-09-15T03:40:59.997272+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:09.173651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,4,1,0,14,13],"acting":[8,2,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.123320+0000","last_change":"2024-09-15T03:40:49.042892+0000","last_active":"2024-09-15T03:40:59.123320+0000","last_peered":"2024-09-15T03:40:59.123320+0000","last_clean":"2024-09-15T03:40:59.123320+0000","last_became_active":"2024-09-15T03:40:46.656459+0000","last_became_peered":"2024-09-15T03:40:46.656459+0000","last_unstale":"2024-09-15T03:40:59.123320+0000","last_undegraded":"2024-09-15T03:40:59.123320+0000","last_fullsized":"2024-09-15T03:40:59.123320+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:52.988670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"35'9","reported_seq":28,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:41:02.144929+0000","last_change":"2024-09-15T03:40:59.756838+0000","last_active":"2024-09-15T03:41:02.144929+0000","last_peered":"2024-09-15T03:41:02.144929+0000","last_clean":"2024-09-15T03:41:02.144929+0000","last_became_active":"2024-09-15T03:40:59.755648+0000","last_became_peered":"2024-09-15T03:40:59.755648+0000","last_unstale":"2024-09-15T03:41:02.144929+0000","last_undegraded":"2024-09-15T03:41:02.144929+0000","last_fullsized":"2024-09-15T03:41:02.144929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.644096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1818624,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":28,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.997372+0000","last_change":"2024-09-15T03:40:48.669967+0000","last_active":"2024-09-15T03:40:59.997372+0000","last_peered":"2024-09-15T03:40:59.997372+0000","last_clean":"2024-09-15T03:40:59.997372+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:40:59.997372+0000","last_undegraded":"2024-09-15T03:40:59.997372+0000","last_fullsized":"2024-09-15T03:40:59.997372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:53.746377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"38'10","reported_seq":34,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:41:05.045159+0000","last_change":"2024-09-15T03:41:05.045784+0000","last_active":"2024-09-15T03:41:05.045159+0000","last_peered":"2024-09-15T03:41:05.045159+0000","last_clean":"2024-09-15T03:41:05.045159+0000","last_became_active":"2024-09-15T03:40:59.756099+0000","last_became_peered":"2024-09-15T03:40:59.756099+0000","last_unstale":"2024-09-15T03:41:05.045159+0000","last_undegraded":"2024-09-15T03:41:05.045159+0000","last_fullsized":"2024-09-15T03:41:05.045159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:09.779018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072176489999999996,"stat_sum":{"num_bytes":1458176,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":7,"num_write_kb":1424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:40:59.001378+0000","last_change":"2024-09-15T03:40:50.313338+0000","last_active":"2024-09-15T03:40:59.001378+0000","last_peered":"2024-09-15T03:40:59.001378+0000","last_clean":"2024-09-15T03:40:59.001378+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:40:59.001378+0000","last_undegraded":"2024-09-15T03:40:59.001378+0000","last_fullsized":"2024-09-15T03:40:59.001378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:58.126050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":17399808,"num_objects":36,"num_object_clones":1,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":79,"num_read_kb":39,"num_write":108,"num_write_kb":18096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":28352512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298264,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073576,"statfs":{"total":96636764160,"available":96331341824,"internally_reserved":0,"allocated":929792,"data_stored":708784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280519,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300308,"kb_used_data":2948,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071532,"statfs":{"total":96636764160,"available":96329248768,"internally_reserved":0,"allocated":3018752,"data_stored":2795041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]}]},{"osd":13,"up_from":22,"seq":94489280518,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300192,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071648,"statfs":{"total":96636764160,"available":96329367552,"internally_reserved":0,"allocated":2899968,"data_stored":2679705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313224,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301332,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070508,"statfs":{"total":96636764160,"available":96328200192,"internally_reserved":0,"allocated":4063232,"data_stored":4082406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":397312,"data_stored":179856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,4,5,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":15,"up_from":21,"seq":90194313224,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297740,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074100,"statfs":{"total":96636764160,"available":96331878400,"internally_reserved":0,"allocated":393216,"data_stored":179278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":3,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297748,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074092,"statfs":{"total":96636764160,"available":96331870208,"internally_reserved":0,"allocated":401408,"data_stored":179875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,7,9,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":397312,"data_stored":179856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]}]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":397312,"data_stored":179856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":397312,"data_stored":179297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297740,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074100,"statfs":{"total":96636764160,"available":96331878400,"internally_reserved":0,"allocated":393216,"data_stored":179278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":4,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":397312,"data_stored":179856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299968,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071872,"statfs":{"total":96636764160,"available":96329596928,"internally_reserved":0,"allocated":2670592,"data_stored":2450399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,8,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":8,"up_from":22,"seq":94489280519,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":397312,"data_stored":179856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297748,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074092,"statfs":{"total":96636764160,"available":96331870208,"internally_reserved":0,"allocated":397312,"data_stored":179856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]}]},{"osd":10,"up_from":22,"seq":94489280519,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299556,"kb_used_data":2196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072284,"statfs":{"total":96636764160,"available":96330018816,"internally_reserved":0,"allocated":2248704,"data_stored":2024993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3604480,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3002368,"data_stored":3002368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2740224,"data_stored":2740224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1818624,"data_stored":1818624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2596864,"data_stored":2596864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3100672,"data_stored":3100672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2568192,"data_stored":2568192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1835008,"data_stored":1835008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1961984,"data_stored":1961984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":2605056,"data_stored":2605056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:41:07.033 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-15T03:41:07.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-15T03:41:07.104 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:07.103+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:07.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:07.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:07.205 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:07.204+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1968174469999999, 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008387205, 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1799999999999999e-07 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017597+0000", 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5058999999999998e-05 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017630+0000", 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3037999999999998e-05 2024-09-15T03:41:07.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017636+0000", 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.254e-06 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017682+0000", 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6839000000000003e-05 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017691+0000", 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3990000000000004e-06 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025936+0000", 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082456730000000002 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025959+0000", 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2725e-05 2024-09-15T03:41:07.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=0, reqid=client.4562.0:13, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1947010950000001, 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011510825000000001, 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019689+0000", 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5799999999999998e-07 2024-09-15T03:41:07.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019738+0000", 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8652000000000001e-05 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019844+0000", 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106623 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019852+0000", 2024-09-15T03:41:07.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019952+0000", 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100111 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019961+0000", 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031174+0000", 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011212744 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031199+0000", 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5278000000000001e-05 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1946394969999998, 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011454499999999999, 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019805+0000", 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5062000000000002e-05 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020203+0000", 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039759399999999997 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020209+0000", 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6189999999999999e-06 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020263+0000", 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4410000000000003e-05 2024-09-15T03:41:07.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020283+0000", 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0176000000000001e-05 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031185+0000", 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010901569 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031204+0000", 2024-09-15T03:41:07.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9663000000000001e-05 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=2, reqid=client.4562.0:14, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1938350199999999, 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010659454, 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020555+0000", 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-07 2024-09-15T03:41:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020569+0000", 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3984e-05 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020610+0000", 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1411e-05 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020616+0000", 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1809999999999998e-06 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022402+0000", 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001785972 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022411+0000", 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6920000000000008e-06 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031203+0000", 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087921389999999992 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031214+0000", 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0902e-05 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021496+0000", 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1928931650000001, 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097223729999999994, 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021496+0000", 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021496+0000", 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021497+0000", 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3499999999999996e-07 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021512+0000", 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5534999999999999e-05 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021542+0000", 2024-09-15T03:41:07.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9068e-05 2024-09-15T03:41:07.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021548+0000", 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9439999999999999e-06 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022379+0000", 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083046499999999996 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022388+0000", 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7050000000000005e-06 2024-09-15T03:41:07.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031208+0000", 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008820095 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031219+0000", 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0926e-05 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1927916380000001, 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012719667, 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021599+0000", 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021655+0000", 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6644999999999997e-05 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021703+0000", 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7549999999999997e-05 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021711+0000", 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1519999999999992e-06 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022571+0000", 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086020699999999999 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022577+0000", 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3339999999999998e-06 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034283+0000", 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011705224 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034317+0000", 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.485e-05 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1920692080000004, 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012060876, 2024-09-15T03:41:07.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022321+0000", 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023195+0000", 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087440700000000001 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023229+0000", 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3905000000000001e-05 2024-09-15T03:41:07.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023237+0000", 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.43e-06 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023278+0000", 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1137999999999998e-05 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023286+0000", 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5379999999999997e-06 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034356+0000", 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01107022 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034381+0000", 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5395999999999999e-05 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.190908211, 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010863246999999999, 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:07.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023482+0000", 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023498+0000", 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6399e-05 2024-09-15T03:41:07.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023536+0000", 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7944000000000001e-05 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023543+0000", 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023580+0000", 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7379999999999998e-05 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023588+0000", 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3950000000000001e-06 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034332+0000", 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010743068 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034344+0000", 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2918000000000001e-05 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1906439320000004, 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010650463000000001, 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023761+0000", 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5421e-05 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023792+0000", 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0366999999999999e-05 2024-09-15T03:41:07.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023797+0000", 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4809999999999997e-06 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025152+0000", 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013547800000000001 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025160+0000", 2024-09-15T03:41:07.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.52e-06 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034385+0000", 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092252549999999999 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034396+0000", 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1161999999999999e-05 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1902245499999999, 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010190506, 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:07.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9900000000000001e-07 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024225+0000", 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9503000000000003e-05 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024258+0000", 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2916999999999997e-05 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024264+0000", 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8900000000000004e-06 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024305+0000", 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1417e-05 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024312+0000", 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0110000000000003e-06 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034347+0000", 2024-09-15T03:41:07.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010034405 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034355+0000", 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8640000000000002e-06 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=7, reqid=client.4562.0:35, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024918+0000", 2024-09-15T03:41:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1894711180000002, 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0094457580000000003, 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024918+0000", 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024918+0000", 2024-09-15T03:41:07.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024919+0000", 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024935+0000", 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5875e-05 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024982+0000", 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7382000000000001e-05 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024991+0000", 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6929999999999996e-06 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025476+0000", 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048515 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025483+0000", 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3839999999999999e-06 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034357+0000", 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088736160000000008 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034364+0000", 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9890000000000003e-06 2024-09-15T03:41:07.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.189156595, 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091838159999999992, 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8699999999999999e-07 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025404+0000", 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017084199999999999 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025430+0000", 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5649000000000001e-05 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025436+0000", 2024-09-15T03:41:07.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0610000000000001e-06 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025486+0000", 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0744000000000001e-05 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025495+0000", 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5019999999999992e-06 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034403+0000", 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089080349999999999 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034417+0000", 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3796e-05 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.273 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1880564290000004, 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018217534000000001, 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026334+0000", 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8500000000000001e-07 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026371+0000", 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6835999999999998e-05 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010326484 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036758+0000", 2024-09-15T03:41:07.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0989999999999997e-05 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036813+0000", 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050000000000003e-05 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036822+0000", 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3149999999999998e-06 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044526+0000", 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077039439999999999 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.423e-05 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.278 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.7s1 35/32 ECSubWrite(tid=2, reqid=client.4562.0:40, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028195+0000", 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1861940149999999, 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089671639999999997, 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028195+0000", 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028195+0000", 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028196+0000", 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028359+0000", 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016298300000000001 2024-09-15T03:41:07.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028390+0000", 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0930999999999999e-05 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028397+0000", 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0690000000000004e-06 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028462+0000", 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028472+0000", 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2739999999999996e-06 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037115+0000", 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086439510000000004 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037163+0000", 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7141000000000001e-05 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.as5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1856475399999997, 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083587509999999993, 2024-09-15T03:41:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.98e-07 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028890+0000", 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014750000000000001 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028929+0000", 2024-09-15T03:41:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9434999999999998e-05 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028937+0000", 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6009999999999997e-06 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029044+0000", 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107129 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029051+0000", 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7060000000000004e-06 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037067+0000", 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080151140000000003 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037101+0000", 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4068e-05 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.287 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.as5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1854968579999996, 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008228997, 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000002e-07 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028902+0000", 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.755e-06 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029308+0000", 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000406678 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029314+0000", 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3140000000000003e-06 2024-09-15T03:41:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031605+0000", 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022915790000000002 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031613+0000", 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8469999999999996e-06 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037103+0000", 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054902939999999997 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037122+0000", 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8108999999999999e-05 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:07.292 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.036286+0000", 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1781035190000004, 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083014009999999999, 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036286+0000", 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036286+0000", 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036287+0000", 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036654+0000", 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036737500000000003 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037195+0000", 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054084700000000001 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037201+0000", 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037245+0000", 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3751000000000002e-05 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037253+0000", 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7700000000000007e-06 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044563+0000", 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073100409999999998 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044587+0000", 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3875000000000001e-05 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:07.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1772249019999999, 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087714329999999993, 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037181+0000", 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5954999999999999e-05 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122876+0000", 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085695108000000006 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122883+0000", 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4069999999999998e-06 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122927+0000", 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552999999999998e-05 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122937+0000", 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.006e-05 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124855+0000", 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001917687 2024-09-15T03:41:07.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124879+0000", 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4002000000000002e-05 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 35/32 ECSubWrite(tid=5, reqid=client.4562.0:61, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.044550+0000", 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1698394109999999, 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008671659, 2024-09-15T03:41:07.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044550+0000", 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044550+0000", 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1800000000000005e-07 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044834+0000", 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028302200000000002 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044868+0000", 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4654999999999999e-05 2024-09-15T03:41:07.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044875+0000", 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7499999999999997e-06 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044917+0000", 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1835000000000002e-05 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044926+0000", 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0589999999999992e-06 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053178+0000", 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082515959999999999 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053222+0000", 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4023999999999997e-05 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 35/32 ECSubWrite(tid=6, reqid=client.4562.0:62, at_version=35'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.1689988390000003, 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007849824, 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:07.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7400000000000003e-07 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045403+0000", 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1839e-05 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045428+0000", 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5097999999999999e-05 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045434+0000", 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7490000000000004e-06 2024-09-15T03:41:07.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047343+0000", 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001908699 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047351+0000", 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7749999999999998e-06 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053230+0000", 2024-09-15T03:41:07.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005879067 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053240+0000", 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0023e-05 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.311 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:07.313 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:07.311+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:07.410 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:07.410 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:07.410 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.414 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:07.413+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:07.418 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:07.514 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:07.514+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:07.516 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:07.516 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:07.516 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:07.517 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:07.615 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:07.615+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:07.664 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6429355880000003, 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008773889, 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019951+0000", 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7080000000000002e-05 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020259+0000", 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030882000000000001 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020269+0000", 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4690000000000003e-06 2024-09-15T03:41:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020353+0000", 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4121000000000003e-05 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020367+0000", 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.401e-05 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028603+0000", 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082364340000000008 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028667+0000", 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3665999999999997e-05 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.669 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6420883110000002, 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075521720000000002, 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020741+0000", 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020828+0000", 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7108999999999996e-05 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020883+0000", 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4818000000000003e-05 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020888+0000", 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.763e-06 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020921+0000", 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2780000000000001e-05 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020926+0000", 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8359999999999997e-06 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028240+0000", 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073138810000000004 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028293+0000", 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2281000000000003e-05 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6419964609999997, 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075072999999999997, 2024-09-15T03:41:07.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020833+0000", 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4400000000000001e-07 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020843+0000", 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020899+0000", 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6416000000000003e-05 2024-09-15T03:41:07.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020905+0000", 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4670000000000003e-06 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022364+0000", 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001459606 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022373+0000", 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5229999999999995e-06 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028305+0000", 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059320400000000004 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028340+0000", 2024-09-15T03:41:07.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4603000000000001e-05 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022913+0000", 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6399160100000003, 2024-09-15T03:41:07.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057677889999999997, 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022913+0000", 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022913+0000", 2024-09-15T03:41:07.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022914+0000", 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0410000000000001e-06 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022933+0000", 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9323000000000001e-05 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022975+0000", 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2311000000000002e-05 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022985+0000", 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9739999999999997e-06 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023030+0000", 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4672999999999997e-05 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023039+0000", 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6319999999999997e-06 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028670+0000", 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056314069999999997 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028681+0000", 2024-09-15T03:41:07.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0428000000000001e-05 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6383581730000003, 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072410060000000004, 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:07.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.74e-07 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024570+0000", 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9729999999999996e-05 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024599+0000", 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8530999999999998e-05 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024605+0000", 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2280000000000003e-06 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024648+0000", 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2466000000000003e-05 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024656+0000", 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8690000000000004e-06 2024-09-15T03:41:07.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031688+0000", 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070326130000000001 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031712+0000", 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3394999999999999e-05 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=4, reqid=client.4562.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6374139359999997, 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063223339999999998, 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025416+0000", 2024-09-15T03:41:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4600000000000004e-07 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025430+0000", 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4636e-05 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025456+0000", 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5718e-05 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025462+0000", 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025538+0000", 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5916999999999998e-05 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025548+0000", 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9620000000000005e-06 2024-09-15T03:41:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031716+0000", 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061672910000000001 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031737+0000", 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1569999999999998e-05 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6371136860000002, 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061423019999999997, 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025716+0000", 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5199999999999996e-07 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025913+0000", 2024-09-15T03:41:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000197252 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025939+0000", 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.552e-05 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025944+0000", 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7320000000000003e-06 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026100+0000", 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575499999999999 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026109+0000", 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3700000000000001e-06 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031831+0000", 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005721271 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031857+0000", 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.675e-05 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026355+0000", 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6364738880000003, 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005535936, 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026355+0000", 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026355+0000", 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026356+0000", 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026372+0000", 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6211000000000001e-05 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026412+0000", 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0034999999999999e-05 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026423+0000", 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0923e-05 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016739529999999999 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028105+0000", 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031878+0000", 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00377298 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031891+0000", 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2584e-05 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=7, reqid=client.4562.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033621+0000", 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6292081669999998, 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003152349, 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033621+0000", 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033621+0000", 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033621+0000", 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5899999999999996e-07 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033631+0000", 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3989999999999993e-06 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033655+0000", 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389e-05 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033662+0000", 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8600000000000004e-06 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033699+0000", 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7150000000000002e-05 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033704+0000", 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3510000000000001e-06 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036757+0000", 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030528059999999999 2024-09-15T03:41:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036773+0000", 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5735000000000001e-05 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6264576220000002, 2024-09-15T03:41:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059101470000000001, 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036372+0000", 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036625+0000", 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025348599999999998 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036653+0000", 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8116e-05 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7190000000000007e-06 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036702+0000", 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0129000000000002e-05 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036711+0000", 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042262+0000", 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005551177 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042281+0000", 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9004e-05 2024-09-15T03:41:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6256427349999996, 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051090550000000004, 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037200+0000", 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3396e-05 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037226+0000", 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6001000000000001e-05 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037231+0000", 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5130000000000004e-06 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037300+0000", 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8978000000000003e-05 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037309+0000", 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6230000000000003e-06 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042287+0000", 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049783509999999998 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042295+0000", 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.039505+0000", 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6233237110000003, 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056020430000000001, 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039505+0000", 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039505+0000", 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039506+0000", 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6899999999999996e-07 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039678+0000", 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172604 2024-09-15T03:41:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039710+0000", 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1216000000000001e-05 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039716+0000", 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1009999999999997e-06 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039760+0000", 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4122000000000002e-05 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039768+0000", 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5290000000000008e-06 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045086+0000", 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053174069999999997 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045107+0000", 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1294999999999999e-05 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/32 ECSubWrite(tid=7, reqid=client.4562.0:56, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.041278+0000", 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.621550601, 2024-09-15T03:41:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038484719999999999, 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041278+0000", 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041278+0000", 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041279+0000", 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4499999999999997e-07 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041291+0000", 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2024e-05 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041315+0000", 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3745000000000001e-05 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041319+0000", 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0260000000000001e-06 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041350+0000", 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1723000000000001e-05 2024-09-15T03:41:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041356+0000", 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2549999999999997e-06 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045115+0000", 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003759685 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045127+0000", 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1369000000000001e-05 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=16, reqid=client.4562.0:118, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.107844+0000", 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.554984889, 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003797749, 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.107844+0000", 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.107844+0000", 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.107845+0000", 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2099999999999995e-07 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108237+0000", 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039230100000000001 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108268+0000", 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0599000000000003e-05 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108274+0000", 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7460000000000003e-06 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108333+0000", 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8322999999999997e-05 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108342+0000", 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:41:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.111620+0000", 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032782419999999998 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.111642+0000", 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1348e-05 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.108390+0000", 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5544392079999998, 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034682089999999999, 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108390+0000", 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108390+0000", 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108390+0000", 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3799999999999997e-07 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108407+0000", 2024-09-15T03:41:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6603e-05 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108432+0000", 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5221000000000001e-05 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108438+0000", 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.626e-06 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108749+0000", 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031144899999999999 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108755+0000", 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0070000000000003e-06 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.111841+0000", 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030857670000000001 2024-09-15T03:41:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.111858+0000", 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6997999999999999e-05 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:124, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5461790410000003, 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043267160000000004, 2024-09-15T03:41:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.99e-07 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117002+0000", 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000351088 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117026+0000", 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4428999999999999e-05 2024-09-15T03:41:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117032+0000", 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5940000000000001e-06 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117074+0000", 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2617000000000002e-05 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117084+0000", 2024-09-15T03:41:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0237e-05 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120955+0000", 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038704239999999999 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120976+0000", 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1628e-05 2024-09-15T03:41:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:125, at_version=35'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.117220+0000", 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.545608477, 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037815679999999999, 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117220+0000", 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117220+0000", 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117221+0000", 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.06e-07 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117237+0000", 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6339999999999999e-05 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117261+0000", 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4043e-05 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117267+0000", 2024-09-15T03:41:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.976e-06 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117468+0000", 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020087699999999999 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117474+0000", 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3520000000000003e-06 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120988+0000", 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003513253 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121002+0000", 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4221e-05 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=18, reqid=client.4562.0:53, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125404+0000", 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5374247929999996, 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003636181, 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125404+0000", 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125404+0000", 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125405+0000", 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125412+0000", 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4889999999999998e-06 2024-09-15T03:41:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125432+0000", 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9694000000000002e-05 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125438+0000", 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1310000000000003e-06 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125471+0000", 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2870000000000002e-05 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125477+0000", 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4150000000000004e-06 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.129018+0000", 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035412289999999999 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.129040+0000", 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpRead(3.cs3 35/32 ECSubRead(tid=19, to_read={3:35272f51:::smithi10536496-18:head=0,4096,0}, subchunks={3:35272f51:::smithi10536496-18:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.126092+0000", 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5367363000000003, 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029229759999999999, 2024-09-15T03:41:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126092+0000", 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126092+0000", 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126093+0000", 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126102+0000", 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3380000000000005e-06 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126133+0000", 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1300000000000002e-05 2024-09-15T03:41:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126140+0000", 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.126174+0000", 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4113999999999999e-05 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.129015+0000", 2024-09-15T03:41:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028412089999999999 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=30, reqid=client.4562.0:171, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.195745+0000", 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4670839920000001, 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031238260000000001, 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195745+0000", 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195745+0000", 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195745+0000", 2024-09-15T03:41:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195760+0000", 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4771e-05 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195787+0000", 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6313999999999999e-05 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195792+0000", 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6690000000000003e-06 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195822+0000", 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0148e-05 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.195827+0000", 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6720000000000003e-06 2024-09-15T03:41:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.198845+0000", 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030179730000000002 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.198869+0000", 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3632e-05 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:07.759 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.760 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:07.758+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:07.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:07.800 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:07.858 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.858 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:07.858 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:07.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.861 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:07.859+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:07.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:07.931 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:07.931 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:07.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:07.962 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:07.961+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:07.964 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:07.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:07.964 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:07.964 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:07.965 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:07.973 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:07.974 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:07.995 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:07.995 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:08.014 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:08.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:set pool 3 pg_num to 26 2024-09-15T03:41:08.035 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:08.035 INFO:tasks.rados.rados.0.smithi105.stdout:77: done (0 left) 2024-09-15T03:41:08.036 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 5 current snap is 3 2024-09-15T03:41:08.036 INFO:tasks.rados.rados.0.smithi105.stdout:78: seq_num 60 ranges {81920=638976,720896=376832} 2024-09-15T03:41:08.039 INFO:tasks.rados.rados.0.smithi105.stdout:78: writing smithi10536496-5 from 81920 to 720896 tid 1 2024-09-15T03:41:08.041 INFO:tasks.rados.rados.0.smithi105.stdout:78: writing smithi10536496-5 from 720896 to 1097728 tid 2 2024-09-15T03:41:08.041 INFO:tasks.rados.rados.0.smithi105.stdout:79: read oid 29 snap -1 2024-09-15T03:41:08.042 INFO:tasks.rados.rados.0.smithi105.stdout:79: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-15T03:41:08.042 INFO:tasks.rados.rados.0.smithi105.stdout:80: read oid 18 snap -1 2024-09-15T03:41:08.042 INFO:tasks.rados.rados.0.smithi105.stdout:80: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-15T03:41:08.042 INFO:tasks.rados.rados.0.smithi105.stdout:81: snap_create 2024-09-15T03:41:08.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:41:08.048 INFO:tasks.rados.rados.0.smithi105.stdout:80: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-15T03:41:08.057 INFO:tasks.rados.rados.0.smithi105.stdout:78: finishing write tid 1 to smithi10536496-5 2024-09-15T03:41:08.057 INFO:tasks.rados.rados.0.smithi105.stdout:79: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-15T03:41:08.062 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:08.061+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:08.069 INFO:tasks.rados.rados.0.smithi105.stdout:78: finishing write tid 2 to smithi10536496-5 2024-09-15T03:41:08.069 INFO:tasks.rados.rados.0.smithi105.stdout:78: finishing write tid 3 to smithi10536496-5 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:15 3.0s0 3:092ceaee:::smithi10536496-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.058347554, 2024-09-15T03:41:08.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021232414000000002, 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-09-15T03:41:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5680000000000001e-06 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018678+0000", 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1760000000000003e-06 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018680+0000", 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.243e-06 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020924+0000", 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022433499999999999 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020978+0000", 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4500000000000003e-05 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032096+0000", 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011118304000000001 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032182+0000", 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5785000000000006e-05 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039904+0000", 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077221329999999999 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:24 3.9s0 3:9a083222:::smithi10536496-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.018889+0000", 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0581305580000002, 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016496140999999999, 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 24 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018889+0000", 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018889+0000", 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018889+0000", 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:41:08.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018898+0000", 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7339999999999997e-06 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018905+0000", 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0679999999999999e-06 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018909+0000", 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9020000000000001e-06 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023442+0000", 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045331629999999998 2024-09-15T03:41:08.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023518+0000", 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5469999999999994e-05 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032717+0000", 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091995059999999997 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032796+0000", 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8527000000000002e-05 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.035385+0000", 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025892910000000001 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:27 3.0s0 3:0304afdb:::smithi10536496-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0578891559999999, 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028611145000000001, 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 27 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019131+0000", 2024-09-15T03:41:08.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019132+0000", 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7460000000000001e-06 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019139+0000", 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019143+0000", 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6720000000000003e-06 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023415+0000", 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042719689999999996 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023491+0000", 2024-09-15T03:41:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5638999999999999e-05 2024-09-15T03:41:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034815+0000", 2024-09-15T03:41:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011324555 2024-09-15T03:41:08.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034893+0000", 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7682999999999995e-05 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047741+0000", 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01284832 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0550359599999997, 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:41:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:41:08.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0484287129999998, 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:41:08.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:41:08.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:08.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.112 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.048411099, 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:41:08.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:41:08.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0436858940000002, 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:41:08.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:64 3.0s0 3:0215fa29:::smithi10536496-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0314973140000001, 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.023494108999999999, 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 64 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045523+0000", 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.358e-06 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045894+0000", 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037004799999999999 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045904+0000", 2024-09-15T03:41:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456e-05 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045909+0000", 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6029999999999998e-06 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045938+0000", 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9054e-05 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046014+0000", 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6334000000000005e-05 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046523+0000", 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000508903 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050383+0000", 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003860086 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068971+0000", 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018587939000000001 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.069016+0000", 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5327999999999997e-05 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.03110415, 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:41:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:41:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:41:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0310926079999998, 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:41:08.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:41:08.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpReadReply(3.0s0 35/32 ECSubReadReply(tid=14, attrs_read=0))", 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.047275+0000", 2024-09-15T03:41:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0297440010000001, 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020799881999999999, 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047275+0000", 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047275+0000", 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047276+0000", 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047286+0000", 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7410000000000004e-06 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067915+0000", 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020629373999999999 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067919+0000", 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5580000000000001e-06 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067946+0000", 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7293000000000001e-05 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068075+0000", 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012929900000000001 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=15, last_complete=35'7, committed=1, applied=1))", 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.048660+0000", 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.028359032, 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020363438000000001, 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.142 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048660+0000", 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048660+0000", 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048661+0000", 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.34e-07 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048664+0000", 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7609999999999999e-06 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068899+0000", 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020235577000000001 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068903+0000", 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0620000000000002e-06 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068926+0000", 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2833999999999999e-05 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.069024+0000", 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7769999999999994e-05 2024-09-15T03:41:08.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0119009810000001, 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.146 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:41:08.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0114170859999998, 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:41:08.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:41:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0113754430000004, 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:41:08.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:41:08.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.162 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0091219960000002, 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:41:08.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:08.166 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.008133655, 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:41:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.9517533050000004, 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036705979, 2024-09-15T03:41:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125267+0000", 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125270+0000", 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161517+0000", 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036246426999999998 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161523+0000", 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161562+0000", 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8571000000000001e-05 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161972+0000", 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041002299999999997 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.9156512059999997, 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:41:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:41:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.9116835420000005, 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:08.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:08.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:08.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:08.184 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:08.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:08.185 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:08.184+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:08.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:08.187 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:08.194 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:08.214 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:08.215 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:08.286 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:08.286+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.305 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:41:08.359 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:08.360 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:08.379 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:08.379 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:08.388 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:08.387+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:08.398 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:08.403 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.403 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.412 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:41:08.418 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:08.422+0000 7fd1c5b38640 0 -- 172.21.15.5:0/651495659 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7fd1c0195170 msgr2=0x7fd1c0197550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:08.443 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:08.463 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:08.489 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:08.487+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:08.509 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.509 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.517 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:08.590 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:08.590+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:08.616 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.616 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.624 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:08.692 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:08.691+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:08.721 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.721 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.729 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:08.775 INFO:tasks.rados.rados.0.smithi105.stdout:78: finishing write tid 4 to smithi10536496-5 2024-09-15T03:41:08.775 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 13 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-15T03:41:08.775 INFO:tasks.rados.rados.0.smithi105.stdout:78: left oid 5 (ObjNum 60 snap 3 seq_num 60) 2024-09-15T03:41:08.793 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:08.792+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.834 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:08.894 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:08.894+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:08.971 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:08.972 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:08.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:08.995 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:08.994+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:08.998 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:41:08.998 INFO:tasks.rados.rados.0.smithi105.stdout:78: done (3 left) 2024-09-15T03:41:08.998 INFO:tasks.rados.rados.0.smithi105.stdout:79: done (2 left) 2024-09-15T03:41:08.998 INFO:tasks.rados.rados.0.smithi105.stdout:80: done (1 left) 2024-09-15T03:41:08.998 INFO:tasks.rados.rados.0.smithi105.stdout:81: done (0 left) 2024-09-15T03:41:08.999 INFO:tasks.rados.rados.0.smithi105.stdout:82: snap_remove snap 3 2024-09-15T03:41:09.079 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.079 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.088 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:09.129 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:09.128+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:09.197 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:09.196+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:09.225 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.226 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.234 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:09.298 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:09.296+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:09.372 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.372 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.380 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:09.398 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:09.397+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:09.477 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.477 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.485 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:09.530 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:09.529+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:09.584 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.584 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:09.639 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:09.638+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:09.701 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:09.700+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:09.708 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.708 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:09.802 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:09.801+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:09.861 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.861 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:09.903 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:09.903+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:09.971 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:09.971 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:09.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:10.000 INFO:tasks.rados.rados.0.smithi105.stdout:82: done (0 left) 2024-09-15T03:41:10.000 INFO:tasks.rados.rados.0.smithi105.stdout:83: copy_from oid 15 from oid 45 current snap is 4 2024-09-15T03:41:10.000 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 47 current snap is 4 2024-09-15T03:41:10.000 INFO:tasks.rados.rados.0.smithi105.stdout:84: seq_num 61 ranges {786432=655360,1441792=376832} 2024-09-15T03:41:10.003 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:10.003+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:10.008 INFO:tasks.rados.rados.0.smithi105.stdout:84: writing smithi10536496-47 from 786432 to 1441792 tid 1 2024-09-15T03:41:10.010 INFO:tasks.rados.rados.0.smithi105.stdout:84: writing smithi10536496-47 from 1441792 to 1818624 tid 2 2024-09-15T03:41:10.010 INFO:tasks.rados.rados.0.smithi105.stdout:85: setattr oid 24 current snap is 4 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:86: read oid 3 snap -1 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:86: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:87: read oid 36 snap -1 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:87: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:88: rollback oid 42 current snap is 4 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:88: done (5 left) 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:89: rollback oid 22 current snap is 4 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:89: done (5 left) 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:90: read oid 50 snap -1 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:90: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-15T03:41:10.012 INFO:tasks.rados.rados.0.smithi105.stdout:91: delete oid 43 current snap is 4 2024-09-15T03:41:10.018 INFO:tasks.rados.rados.0.smithi105.stdout:83: finishing copy_from to smithi10536496-15 2024-09-15T03:41:10.018 INFO:tasks.rados.rados.0.smithi105.stdout:83: got expected ENOENT (src dne) 2024-09-15T03:41:10.018 INFO:tasks.rados.rados.0.smithi105.stdout:83: finishing copy_from racing read to smithi10536496-15 2024-09-15T03:41:10.020 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 7 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-15T03:41:10.023 INFO:tasks.rados.rados.0.smithi105.stdout:84: finishing write tid 1 to smithi10536496-47 2024-09-15T03:41:10.023 INFO:tasks.rados.rados.0.smithi105.stdout:84: finishing write tid 2 to smithi10536496-47 2024-09-15T03:41:10.024 INFO:tasks.rados.rados.0.smithi105.stdout:84: finishing write tid 3 to smithi10536496-47 2024-09-15T03:41:10.027 INFO:tasks.rados.rados.0.smithi105.stdout:90: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-15T03:41:10.051 INFO:tasks.rados.rados.0.smithi105.stdout:84: finishing write tid 4 to smithi10536496-47 2024-09-15T03:41:10.051 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 10 (ObjNum 61 snap 4 seq_num 61) dirty exists 2024-09-15T03:41:10.051 INFO:tasks.rados.rados.0.smithi105.stdout:84: left oid 47 (ObjNum 61 snap 4 seq_num 61) 2024-09-15T03:41:10.051 INFO:tasks.rados.rados.0.smithi105.stdout:86: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:83: done (6 left) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:84: done (5 left) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:85: done (4 left) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:86: done (3 left) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:90: done (2 left) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:91: done (1 left) 2024-09-15T03:41:10.078 INFO:tasks.rados.rados.0.smithi105.stdout:92: rollback oid 29 current snap is 4 2024-09-15T03:41:10.079 INFO:tasks.rados.rados.0.smithi105.stdout:92: done (1 left) 2024-09-15T03:41:10.079 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 42 current snap is 4 2024-09-15T03:41:10.079 INFO:tasks.rados.rados.0.smithi105.stdout:93: seq_num 63 ranges {704512=688128,1392640=376832} 2024-09-15T03:41:10.085 INFO:tasks.rados.rados.0.smithi105.stdout:93: writing smithi10536496-42 from 704512 to 1392640 tid 1 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:93: writing smithi10536496-42 from 1392640 to 1769472 tid 2 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:94: rollback oid 46 current snap is 4 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:94: done (2 left) 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:95: copy_from oid 10 from oid 27 current snap is 4 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:96: read oid 2 snap -1 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:96: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-15T03:41:10.088 INFO:tasks.rados.rados.0.smithi105.stdout:97: setattr oid 49 current snap is 4 2024-09-15T03:41:10.090 INFO:tasks.rados.rados.0.smithi105.stdout:98: read oid 47 snap -1 2024-09-15T03:41:10.090 INFO:tasks.rados.rados.0.smithi105.stdout:98: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-15T03:41:10.091 INFO:tasks.rados.rados.0.smithi105.stdout:99: delete oid 22 current snap is 4 2024-09-15T03:41:10.091 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.091 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.092 INFO:tasks.rados.rados.0.smithi105.stdout:93: finishing write tid 1 to smithi10536496-42 2024-09-15T03:41:10.094 INFO:tasks.rados.rados.0.smithi105.stdout:96: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-15T03:41:10.104 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:10.105 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:10.103+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:10.119 INFO:tasks.rados.rados.0.smithi105.stdout:93: finishing write tid 2 to smithi10536496-42 2024-09-15T03:41:10.119 INFO:tasks.rados.rados.0.smithi105.stdout:95: finishing copy_from racing read to smithi10536496-10 2024-09-15T03:41:10.119 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 22 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-15T03:41:10.119 INFO:tasks.rados.rados.0.smithi105.stdout:96: done (6 left) 2024-09-15T03:41:10.119 INFO:tasks.rados.rados.0.smithi105.stdout:97: done (5 left) 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:99: done (4 left) 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:100: read oid 28 snap -1 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:100: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:101: copy_from oid 38 from oid 40 current snap is 4 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:95: finishing copy_from to smithi10536496-10 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:95: done (5 left) 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:102: read oid 50 snap -1 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:102: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 18 current snap is 4 2024-09-15T03:41:10.120 INFO:tasks.rados.rados.0.smithi105.stdout:103: seq_num 65 ranges {294912=720896,1015808=376832} 2024-09-15T03:41:10.124 INFO:tasks.rados.rados.0.smithi105.stdout:103: writing smithi10536496-18 from 294912 to 1015808 tid 1 2024-09-15T03:41:10.127 INFO:tasks.rados.rados.0.smithi105.stdout:103: writing smithi10536496-18 from 1015808 to 1392640 tid 2 2024-09-15T03:41:10.127 INFO:tasks.rados.rados.0.smithi105.stdout:104: delete oid 19 current snap is 4 2024-09-15T03:41:10.127 INFO:tasks.rados.rados.0.smithi105.stdout:100: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:10.140 INFO:tasks.rados.rados.0.smithi105.stdout:102: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-15T03:41:10.161 INFO:tasks.rados.rados.0.smithi105.stdout:98: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-15T03:41:10.204 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:10.203+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:10.208 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.209 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.217 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:10.219 INFO:tasks.rados.rados.0.smithi105.stdout:103: finishing write tid 1 to smithi10536496-18 2024-09-15T03:41:10.219 INFO:tasks.rados.rados.0.smithi105.stdout:103: finishing write tid 2 to smithi10536496-18 2024-09-15T03:41:10.219 INFO:tasks.rados.rados.0.smithi105.stdout:98: done (7 left) 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:100: done (6 left) 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:102: done (5 left) 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:104: done (4 left) 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:105: setattr oid 44 current snap is 4 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:106: read oid 43 snap -1 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:106: expect deleted 2024-09-15T03:41:10.220 INFO:tasks.rados.rados.0.smithi105.stdout:107: delete oid 24 current snap is 4 2024-09-15T03:41:10.225 INFO:tasks.rados.rados.0.smithi105.stdout:107: done (6 left) 2024-09-15T03:41:10.225 INFO:tasks.rados.rados.0.smithi105.stdout:108: delete oid 45 current snap is 4 2024-09-15T03:41:10.231 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 17 (ObjNum 58 snap 2 seq_num 58) dirty exists 2024-09-15T03:41:10.231 INFO:tasks.rados.rados.0.smithi105.stdout:105: done (6 left) 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:106: done (5 left) 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:108: done (4 left) 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:109: rollback oid 21 current snap is 4 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:109: done (4 left) 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:110: rmattr oid 20 current snap is 4 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:110: done (4 left) 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:111: copy_from oid 46 from oid 28 current snap is 4 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:112: read oid 14 snap -1 2024-09-15T03:41:10.232 INFO:tasks.rados.rados.0.smithi105.stdout:112: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-15T03:41:10.233 INFO:tasks.rados.rados.0.smithi105.stdout:113: snap_create 2024-09-15T03:41:10.236 INFO:tasks.rados.rados.0.smithi105.stdout:112: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-15T03:41:10.241 INFO:tasks.rados.rados.0.smithi105.stdout:111: finishing copy_from racing read to smithi10536496-46 2024-09-15T03:41:10.242 INFO:tasks.rados.rados.0.smithi105.stdout:87: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-15T03:41:10.305 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:10.305+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:10.329 INFO:tasks.rados.rados.0.smithi105.stdout:101: finishing copy_from racing read to smithi10536496-38 2024-09-15T03:41:10.329 INFO:tasks.rados.rados.0.smithi105.stdout:101: finishing copy_from to smithi10536496-38 2024-09-15T03:41:10.329 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-15T03:41:10.360 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.361 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:10.414 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:10.413+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:10.471 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.471 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:10.475 INFO:tasks.rados.rados.0.smithi105.stdout:111: finishing copy_from to smithi10536496-46 2024-09-15T03:41:10.475 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-15T03:41:10.480 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:10.492 INFO:tasks.rados.rados.0.smithi105.stdout:103: finishing write tid 3 to smithi10536496-18 2024-09-15T03:41:10.493 INFO:tasks.rados.rados.0.smithi105.stdout:103: finishing write tid 4 to smithi10536496-18 2024-09-15T03:41:10.493 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 12 (ObjNum 65 snap 4 seq_num 65) dirty exists 2024-09-15T03:41:10.493 INFO:tasks.rados.rados.0.smithi105.stdout:103: left oid 18 (ObjNum 65 snap 4 seq_num 65) 2024-09-15T03:41:10.507 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:10.507+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:10.604 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:10.608 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:10.607+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:10.624 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.625 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.626 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:10.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:10.647 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:10.668 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:10.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:10.689 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:10.708 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:10.708+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:10.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:10.709 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:10.738 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.738 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:10.827 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:10.827 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:10.825+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:10.847 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:10.847 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:10.867 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:10.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:10.888 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:10.891 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:10.891 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:10.895 INFO:tasks.rados.rados.0.smithi105.stdout:93: finishing write tid 3 to smithi10536496-42 2024-09-15T03:41:10.896 INFO:tasks.rados.rados.0.smithi105.stdout:93: finishing write tid 4 to smithi10536496-42 2024-09-15T03:41:10.896 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 7 (ObjNum 63 snap 4 seq_num 63) dirty exists 2024-09-15T03:41:10.896 INFO:tasks.rados.rados.0.smithi105.stdout:93: left oid 42 (ObjNum 63 snap 4 seq_num 63) 2024-09-15T03:41:10.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:10.910 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:10.911 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:10.908+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:10.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:10.929 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:10.949 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:10.949 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:10.984 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:10.985 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:10.989+0000 7f1cc37fe640 0 -- 172.21.15.5:0/3952665830 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f1cc41954d0 msgr2=0x7f1cc41978b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:87: done (6 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:93: done (5 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:101: done (4 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:103: done (3 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:111: done (2 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:112: done (1 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:113: done (0 left) 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 13 current snap is 5 2024-09-15T03:41:11.006 INFO:tasks.rados.rados.0.smithi105.stdout:114: seq_num 69 ranges {1114112=786432,1900544=376832} 2024-09-15T03:41:11.011 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:11.010+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:11.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:41:11.017 INFO:tasks.rados.rados.0.smithi105.stdout:114: writing smithi10536496-13 from 1114112 to 1900544 tid 1 2024-09-15T03:41:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:114: writing smithi10536496-13 from 1900544 to 2277376 tid 2 2024-09-15T03:41:11.020 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 14 current snap is 5 2024-09-15T03:41:11.020 INFO:tasks.rados.rados.0.smithi105.stdout:115: seq_num 70 ranges {229376=802816,1032192=376832} 2024-09-15T03:41:11.026 INFO:tasks.rados.rados.0.smithi105.stdout:115: writing smithi10536496-14 from 229376 to 1032192 tid 1 2024-09-15T03:41:11.027 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:11.027 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:11.029 INFO:tasks.rados.rados.0.smithi105.stdout:115: writing smithi10536496-14 from 1032192 to 1409024 tid 2 2024-09-15T03:41:11.029 INFO:tasks.rados.rados.0.smithi105.stdout:116: rollback oid 45 current snap is 5 2024-09-15T03:41:11.029 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 45 to 4 2024-09-15T03:41:11.029 INFO:tasks.rados.rados.0.smithi105.stdout:117: snap_create 2024-09-15T03:41:11.031 INFO:tasks.rados.rados.0.smithi105.stdout:114: finishing write tid 1 to smithi10536496-13 2024-09-15T03:41:11.032 INFO:tasks.rados.rados.0.smithi105.stdout:114: finishing write tid 2 to smithi10536496-13 2024-09-15T03:41:11.032 INFO:tasks.rados.rados.0.smithi105.stdout:114: finishing write tid 3 to smithi10536496-13 2024-09-15T03:41:11.033 INFO:tasks.rados.rados.0.smithi105.stdout:114: finishing write tid 4 to smithi10536496-13 2024-09-15T03:41:11.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 16 (ObjNum 69 snap 5 seq_num 69) dirty exists 2024-09-15T03:41:11.033 INFO:tasks.rados.rados.0.smithi105.stdout:114: left oid 13 (ObjNum 69 snap 5 seq_num 69) 2024-09-15T03:41:11.037 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:11.038 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:11.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:11.048 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:11.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:11.111 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:11.111+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:11.155 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:11.155 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:11.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 34/32 ECSubWrite(tid=0, reqid=client.4562.0:4, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017387+0000", 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1343301740000005, 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001891144, 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.156 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017387+0000", 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017387+0000", 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017388+0000", 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.762e-06 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:11.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6598999999999999e-05 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017509+0000", 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3826999999999996e-05 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017522+0000", 2024-09-15T03:41:11.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3577000000000001e-05 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017590+0000", 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7854000000000005e-05 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017602+0000", 2024-09-15T03:41:11.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1707e-05 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019248+0000", 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016461399999999999 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019278+0000", 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9677999999999999e-05 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1333015050000004, 2024-09-15T03:41:11.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023454370000000001, 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018416+0000", 2024-09-15T03:41:11.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5300000000000004e-07 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018457+0000", 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021725800000000001 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018754+0000", 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0282000000000001e-05 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018806+0000", 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018858+0000", 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1873000000000003e-05 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020727+0000", 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001868897 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020761+0000", 2024-09-15T03:41:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4165999999999998e-05 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=1, reqid=client.4562.0:8, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1313606420000006, 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021286080000000001, 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020357+0000", 2024-09-15T03:41:11.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0139999999999999e-06 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020398+0000", 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1109000000000002e-05 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020456+0000", 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7636999999999997e-05 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020465+0000", 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6230000000000003e-06 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020522+0000", 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7314e-05 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020532+0000", 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:11.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022453+0000", 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019207950000000001 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022485+0000", 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1915000000000002e-05 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.172 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.020999+0000", 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1307178009999994, 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015421569999999999, 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020999+0000", 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020999+0000", 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021000+0000", 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4200000000000004e-07 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021055+0000", 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4765999999999997e-05 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021126+0000", 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1644999999999996e-05 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021133+0000", 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6610000000000003e-06 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021188+0000", 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5238000000000002e-05 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021218+0000", 2024-09-15T03:41:11.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022461+0000", 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001243156 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022541+0000", 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9655999999999998e-05 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1306580719999992, 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017195820000000001, 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:11.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-07 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021090+0000", 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0531999999999998e-05 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021617+0000", 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00052774400000000002 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021651+0000", 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3312000000000002e-05 2024-09-15T03:41:11.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021685+0000", 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4638999999999998e-05 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021694+0000", 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.149e-06 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022749+0000", 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010543919999999999 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022778+0000", 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9476e-05 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.182 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028006+0000", 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1237109190000005, 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001802898, 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028006+0000", 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028006+0000", 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028007+0000", 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.28e-06 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028092+0000", 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4579000000000005e-05 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028243+0000", 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015144600000000001 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028257+0000", 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3343000000000001e-05 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028338+0000", 2024-09-15T03:41:11.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1169999999999997e-05 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028359+0000", 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1137000000000001e-05 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029778+0000", 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014188060000000001 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029809+0000", 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1136999999999997e-05 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1236193579999991, 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001736865, 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.187 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028098+0000", 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028119+0000", 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0625999999999999e-05 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028357+0000", 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000238545 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028781+0000", 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00042363600000000002 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028878+0000", 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7279000000000002e-05 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028887+0000", 2024-09-15T03:41:11.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8259999999999999e-06 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029819+0000", 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00093264499999999996 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029834+0000", 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4816e-05 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.191 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1235761279999998, 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020128619999999998, 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:11.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.49e-07 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028149+0000", 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4889999999999995e-06 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028822+0000", 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067247100000000001 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028830+0000", 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1429999999999998e-06 2024-09-15T03:41:11.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0082999999999998e-05 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028930+0000", 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0395e-05 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030125+0000", 2024-09-15T03:41:11.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011949440000000001 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030154+0000", 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8187999999999998e-05 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.036498+0000", 2024-09-15T03:41:11.196 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1152192070000009, 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017902619999999999, 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036498+0000", 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036498+0000", 2024-09-15T03:41:11.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036499+0000", 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.540000000000001e-07 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019873100000000001 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036735+0000", 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7564000000000002e-05 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036742+0000", 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7610000000000002e-06 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036811+0000", 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9587999999999998e-05 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036821+0000", 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0035e-05 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.038256+0000", 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001435183 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.038288+0000", 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1445999999999997e-05 2024-09-15T03:41:11.200 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.037224+0000", 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1144923260000006, 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001472234, 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037224+0000", 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037224+0000", 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037225+0000", 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.71e-07 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037262+0000", 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6349e-05 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037302+0000", 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0179999999999998e-05 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037321+0000", 2024-09-15T03:41:11.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9457999999999999e-05 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037365+0000", 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3680999999999997e-05 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.037383+0000", 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8029e-05 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.038671+0000", 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012880249999999999 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.038697+0000", 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5641000000000001e-05 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.205 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=4, reqid=client.4562.0:58, at_version=35'3, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.041763+0000", 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.1099533130000001, 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001501861, 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.041763+0000", 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.041763+0000", 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.041765+0000", 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.164e-06 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.042007+0000", 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024280300000000001 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.042059+0000", 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1561999999999999e-05 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.042069+0000", 2024-09-15T03:41:11.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0434999999999999e-05 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.042131+0000", 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1344000000000001e-05 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.042142+0000", 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0953e-05 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.043236+0000", 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00109443 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.043265+0000", 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.917e-05 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=4, reqid=client.4562.0:79, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.060879+0000", 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.0908376650000005, 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015866260000000001, 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.060879+0000", 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.060879+0000", 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.060881+0000", 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.64e-06 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061083+0000", 2024-09-15T03:41:11.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000202716 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061134+0000", 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0464999999999999e-05 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061144+0000", 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0389e-05 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061236+0000", 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2149000000000002e-05 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061247+0000", 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0818e-05 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062437+0000", 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001189369 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062466+0000", 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9079999999999999e-05 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.215 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.0807727580000002, 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084043045999999996, 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070945+0000", 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.124e-06 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070959+0000", 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4212999999999999e-05 2024-09-15T03:41:11.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154508+0000", 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083548836000000001 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154515+0000", 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:41:11.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154566+0000", 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1025e-05 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154578+0000", 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154965+0000", 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038690300000000002 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154987+0000", 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2172999999999999e-05 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.220 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=24, reqid=client.4562.0:100, at_version=35'3, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.0682426679999999, 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002110823, 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:11.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083476+0000", 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.807e-06 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083643+0000", 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016675300000000001 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083675+0000", 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1962000000000003e-05 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083684+0000", 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6930000000000002e-06 2024-09-15T03:41:11.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083746+0000", 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2008000000000007e-05 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083756+0000", 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9450000000000005e-06 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085554+0000", 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017979980000000001 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085585+0000", 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0657000000000001e-05 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=33, reqid=client.4562.0:139, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:11.225 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.141221+0000", 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.0104958899999996, 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015497499999999999, 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141221+0000", 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141221+0000", 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141222+0000", 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.37e-06 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141486+0000", 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00026379799999999998 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141511+0000", 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4564000000000001e-05 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141518+0000", 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1199999999999996e-06 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141578+0000", 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9803999999999999e-05 2024-09-15T03:41:11.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141590+0000", 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2024e-05 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142743+0000", 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001153893 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142771+0000", 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7177000000000001e-05 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=12, reqid=client.4562.0:150, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.9974724990000006, 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035676890000000002, 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154245+0000", 2024-09-15T03:41:11.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154487+0000", 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000242308 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156728+0000", 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022401589999999998 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156736+0000", 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3259999999999996e-06 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156781+0000", 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4583000000000003e-05 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156789+0000", 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8359999999999998e-06 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157761+0000", 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097187399999999996 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157812+0000", 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0695999999999997e-05 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=13, reqid=client.4562.0:151, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.156461+0000", 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.9952556789999996, 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018052599999999999, 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156461+0000", 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156461+0000", 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156462+0000", 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.54e-07 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156706+0000", 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024380699999999999 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157225+0000", 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00051943900000000001 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157233+0000", 2024-09-15T03:41:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9310000000000008e-06 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157278+0000", 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5082e-05 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157287+0000", 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4200000000000007e-06 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.158240+0000", 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00095330000000000002 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.158266+0000", 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6526999999999999e-05 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=14, reqid=client.4562.0:152, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.9945286089999996, 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028494564, 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157189+0000", 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3399999999999998e-07 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157206+0000", 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7207e-05 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185008+0000", 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027801691999999999 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185015+0000", 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185068+0000", 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2333000000000002e-05 2024-09-15T03:41:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185077+0000", 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0259999999999996e-06 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185660+0000", 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058343299999999996 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185683+0000", 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714e-05 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=15, reqid=client.4562.0:164, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.9670488400000004, 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027286849999999998, 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184669+0000", 2024-09-15T03:41:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000005e-07 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184986+0000", 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000317027 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015847960000000001 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186578+0000", 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3220000000000004e-06 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186619+0000", 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1186000000000002e-05 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186629+0000", 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:11.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187377+0000", 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074841899999999995 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187397+0000", 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9233999999999999e-05 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=17, reqid=client.4562.0:166, at_version=35'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.9648318580000002, 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099130431000000005, 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186886+0000", 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:41:11.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186906+0000", 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0482000000000001e-05 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285393+0000", 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098487125999999994 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285401+0000", 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3780000000000003e-06 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285446+0000", 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028999999999999e-05 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285453+0000", 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285995+0000", 2024-09-15T03:41:11.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00054192900000000002 2024-09-15T03:41:11.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:11.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:11.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.286015+0000", 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0579000000000001e-05 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:11.257 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:11.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:11.259 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:11.257+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:11.360 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:11.358+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:11.373 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:11.373 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:11.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:11.461 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:11.461+0000 7f2f553f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33103) UID: 0 2024-09-15T03:41:11.484 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:11.484 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:11.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:11.562 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:11.562+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:11.595 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5761522729999999, 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094904819999999997, 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.23e-07 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017537+0000", 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8838000000000003e-05 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017912+0000", 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037478400000000002 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017919+0000", 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8010000000000004e-06 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017987+0000", 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7138000000000007e-05 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017998+0000", 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0908e-05 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026958+0000", 2024-09-15T03:41:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089603229999999992 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026988+0000", 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0467000000000002e-05 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:5, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5754262150000002, 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085434570000000008, 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018225+0000", 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2619999999999999e-06 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018254+0000", 2024-09-15T03:41:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8656000000000001e-05 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018297+0000", 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999999e-05 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018310+0000", 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3052e-05 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018392+0000", 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1527999999999996e-05 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4590000000000004e-06 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026715+0000", 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083134909999999992 2024-09-15T03:41:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026767+0000", 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599000000000002e-05 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5748819449999996, 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080389840000000008, 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1199999999999999e-07 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018811+0000", 2024-09-15T03:41:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2938000000000001e-05 2024-09-15T03:41:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018916+0000", 2024-09-15T03:41:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010431 2024-09-15T03:41:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018923+0000", 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019065+0000", 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014137500000000001 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019075+0000", 2024-09-15T03:41:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0271e-05 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026774+0000", 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076987230000000002 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026807+0000", 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3655999999999998e-05 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5720418410000008, 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010778574000000001, 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021609+0000", 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021692+0000", 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999997e-05 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021735+0000", 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3581000000000002e-05 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021743+0000", 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3119999999999994e-06 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021822+0000", 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8437999999999997e-05 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021830+0000", 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2989999999999997e-06 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032339+0000", 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010508768999999999 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032387+0000", 2024-09-15T03:41:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7851e-05 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5713890569999993, 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010171065999999999, 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022262+0000", 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.043e-06 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022318+0000", 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6208000000000001e-05 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022353+0000", 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4437000000000002e-05 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022362+0000", 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022408+0000", 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5775000000000001e-05 2024-09-15T03:41:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022419+0000", 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1450999999999999e-05 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032351+0000", 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099319260000000006 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032432+0000", 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1032000000000006e-05 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5707215570000006, 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094003279999999995, 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022930+0000", 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022949+0000", 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9916e-05 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022995+0000", 2024-09-15T03:41:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5173000000000003e-05 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023004+0000", 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2990000000000002e-06 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023067+0000", 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023073+0000", 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032300+0000", 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092268320000000008 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032329+0000", 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8779999999999999e-05 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5699858510000002, 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087825990000000003, 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023665+0000", 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5199999999999995e-07 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023681+0000", 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-05 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023713+0000", 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1726000000000001e-05 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023720+0000", 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6830000000000002e-06 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023765+0000", 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4988999999999999e-05 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023771+0000", 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9109999999999998e-06 2024-09-15T03:41:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032437+0000", 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086659760000000006 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032447+0000", 2024-09-15T03:41:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0222e-05 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5699360789999997, 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087050259999999994, 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023715+0000", 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023733+0000", 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8442999999999999e-05 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023770+0000", 2024-09-15T03:41:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6585999999999998e-05 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023784+0000", 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4440999999999999e-05 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023829+0000", 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5099000000000003e-05 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023836+0000", 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4400000000000002e-06 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032396+0000", 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085599330000000005 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032419+0000", 2024-09-15T03:41:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3288000000000001e-05 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5693568720000002, 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081451809999999996, 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024294+0000", 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024371+0000", 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6451000000000004e-05 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024415+0000", 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4292000000000002e-05 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024423+0000", 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0700000000000007e-06 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024479+0000", 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024488+0000", 2024-09-15T03:41:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1109999999999997e-06 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032423+0000", 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079349969999999992 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032438+0000", 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5279000000000001e-05 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5665095079999993, 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013181325000000001, 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7000000000000004e-07 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027155+0000", 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027190+0000", 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5738000000000002e-05 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027199+0000", 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1999999999999994e-06 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027257+0000", 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8690000000000002e-05 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027265+0000", 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8860000000000004e-06 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040256+0000", 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012990323999999999 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040322+0000", 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6392999999999999e-05 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5660308040000004, 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012779128000000001, 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027620+0000", 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.32e-07 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027692+0000", 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000006e-05 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027773+0000", 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0605000000000005e-05 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027782+0000", 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-15T03:41:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027884+0000", 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010173100000000001 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027894+0000", 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9210000000000004e-06 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040357+0000", 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463284999999999 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1026000000000003e-05 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5647430999999994, 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011382537, 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028908+0000", 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0300000000000001e-06 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028922+0000", 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4329e-05 2024-09-15T03:41:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028975+0000", 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2573e-05 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028983+0000", 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029027+0000", 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3912000000000002e-05 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029036+0000", 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9560000000000003e-06 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040254+0000", 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011217632999999999 2024-09-15T03:41:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5880999999999997e-05 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5647303669999992, 2024-09-15T03:41:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011534467, 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028921+0000", 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.057e-06 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028959+0000", 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8217999999999999e-05 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028993+0000", 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4425999999999999e-05 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029011+0000", 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7465e-05 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029270+0000", 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025923899999999998 2024-09-15T03:41:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029280+0000", 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040431+0000", 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011151283 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040454+0000", 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3309e-05 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5640923059999992, 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013707001, 2024-09-15T03:41:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029559+0000", 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029676+0000", 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117273 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029720+0000", 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4026000000000002e-05 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029728+0000", 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3240000000000003e-06 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030022+0000", 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029338200000000002 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030037+0000", 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5082000000000001e-05 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043234+0000", 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013196548000000001 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043265+0000", 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1312000000000001e-05 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5630318659999993, 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097244270000000008, 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030619+0000", 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999998e-07 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030630+0000", 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030662+0000", 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1813000000000002e-05 2024-09-15T03:41:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030680+0000", 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8306999999999999e-05 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030722+0000", 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1823000000000003e-05 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030729+0000", 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040325+0000", 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095952650000000004 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040343+0000", 2024-09-15T03:41:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8079000000000001e-05 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5620403639999996, 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011724098000000001, 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031742+0000", 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013129600000000001 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031779+0000", 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7311000000000002e-05 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031787+0000", 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031842+0000", 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4617999999999998e-05 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031856+0000", 2024-09-15T03:41:11.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.42e-05 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043313+0000", 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011457292000000001 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043334+0000", 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-15T03:41:11.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=9, reqid=client.4562.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5618419219999993, 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011489871, 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031809+0000", 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0130000000000001e-06 2024-09-15T03:41:11.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031830+0000", 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0801999999999998e-05 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031864+0000", 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3575000000000003e-05 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031869+0000", 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5029999999999997e-06 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032118+0000", 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024897299999999999 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032128+0000", 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4879999999999996e-06 2024-09-15T03:41:11.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043276+0000", 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01114878 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043298+0000", 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1736999999999999e-05 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5611713090000006, 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010878669000000001, 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032507+0000", 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7140000000000001e-05 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032542+0000", 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4986e-05 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032549+0000", 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2130000000000002e-06 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034852+0000", 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023031089999999998 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034862+0000", 2024-09-15T03:41:11.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0156000000000001e-05 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043343+0000", 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084812489999999997 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043357+0000", 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4219000000000001e-05 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5585292769999999, 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008115901, 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035122+0000", 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035140+0000", 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7915999999999999e-05 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035176+0000", 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6335999999999999e-05 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035184+0000", 2024-09-15T03:41:11.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035238+0000", 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4048000000000002e-05 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035248+0000", 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043207+0000", 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079593690000000009 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043237+0000", 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9788e-05 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.692 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=20, reqid=client.4562.0:159, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4332316699999996, 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091519410000000002, 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160419+0000", 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7499999999999999e-07 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160707+0000", 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028803500000000002 2024-09-15T03:41:11.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160722+0000", 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5235000000000001e-05 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160727+0000", 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7600000000000002e-06 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:11.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160775+0000", 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7732999999999999e-05 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160785+0000", 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0380999999999999e-05 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169551+0000", 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087657610000000004 2024-09-15T03:41:11.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169570+0000", 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9261000000000001e-05 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:11.697 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:11.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:11.699 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:11.697+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:11.800 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:11.799+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:11.842 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:11.842 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:11.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:11.900 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:11.899+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:11.951 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:11.952 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:11.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:12.002 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:12.001+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:41:12.012 INFO:tasks.rados.rados.0.smithi105.stdout:114: done (3 left) 2024-09-15T03:41:12.012 INFO:tasks.rados.rados.0.smithi105.stdout:117: done (2 left) 2024-09-15T03:41:12.012 INFO:tasks.rados.rados.0.smithi105.stdout:118: copy_from oid 35 from oid 15 current snap is 6 2024-09-15T03:41:12.012 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 40 current snap is 6 2024-09-15T03:41:12.012 INFO:tasks.rados.rados.0.smithi105.stdout:119: seq_num 71 ranges {671744=425984,1097728=573440,1671168=196608} 2024-09-15T03:41:12.018 INFO:tasks.rados.rados.0.smithi105.stdout:119: writing smithi10536496-40 from 671744 to 1097728 tid 1 2024-09-15T03:41:12.022 INFO:tasks.rados.rados.0.smithi105.stdout:119: writing smithi10536496-40 from 1097728 to 1671168 tid 2 2024-09-15T03:41:12.023 INFO:tasks.rados.rados.0.smithi105.stdout:119: writing smithi10536496-40 from 1671168 to 1867776 tid 3 2024-09-15T03:41:12.024 INFO:tasks.rados.rados.0.smithi105.stdout:120: setattr oid 1 current snap is 6 2024-09-15T03:41:12.028 INFO:tasks.rados.rados.0.smithi105.stdout:121: copy_from oid 5 from oid 3 current snap is 6 2024-09-15T03:41:12.028 INFO:tasks.rados.rados.0.smithi105.stdout:122: rollback oid 39 current snap is 6 2024-09-15T03:41:12.028 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 39 to 5 2024-09-15T03:41:12.028 INFO:tasks.rados.rados.0.smithi105.stdout:123: snap_create 2024-09-15T03:41:12.032 INFO:tasks.rados.rados.0.smithi105.stdout:119: finishing write tid 1 to smithi10536496-40 2024-09-15T03:41:12.034 INFO:tasks.rados.rados.0.smithi105.stdout:119: finishing write tid 2 to smithi10536496-40 2024-09-15T03:41:12.038 INFO:tasks.rados.rados.0.smithi105.stdout:119: finishing write tid 3 to smithi10536496-40 2024-09-15T03:41:12.038 INFO:tasks.rados.rados.0.smithi105.stdout:119: finishing write tid 4 to smithi10536496-40 2024-09-15T03:41:12.039 INFO:tasks.rados.rados.0.smithi105.stdout:119: finishing write tid 5 to smithi10536496-40 2024-09-15T03:41:12.039 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 28 (ObjNum 71 snap 6 seq_num 71) dirty exists 2024-09-15T03:41:12.039 INFO:tasks.rados.rados.0.smithi105.stdout:119: left oid 40 (ObjNum 71 snap 6 seq_num 71) 2024-09-15T03:41:12.044 INFO:tasks.rados.rados.0.smithi105.stdout:122: finishing rollback tid 0 to smithi10536496-39 2024-09-15T03:41:12.044 INFO:tasks.rados.rados.0.smithi105.stdout:122: finishing rollback tid 1 to smithi10536496-39 2024-09-15T03:41:12.045 INFO:tasks.rados.rados.0.smithi105.stdout:116: finishing rollback tid 1 to smithi10536496-45 2024-09-15T03:41:12.045 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 129 snap 0 seq_num 3899264232) dirty dne 2024-09-15T03:41:12.047 INFO:tasks.rados.rados.0.smithi105.stdout:122: finishing rollback tid 2 to smithi10536496-39 2024-09-15T03:41:12.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 13 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T03:41:12.054 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 11 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T03:41:12.058 INFO:tasks.rados.rados.0.smithi105.stdout:118: finishing copy_from racing read to smithi10536496-35 2024-09-15T03:41:12.058 INFO:tasks.rados.rados.0.smithi105.stdout:121: finishing copy_from racing read to smithi10536496-5 2024-09-15T03:41:12.059 INFO:tasks.rados.rados.0.smithi105.stdout:115: finishing write tid 1 to smithi10536496-14 2024-09-15T03:41:12.059 INFO:tasks.rados.rados.0.smithi105.stdout:115: finishing write tid 2 to smithi10536496-14 2024-09-15T03:41:12.059 INFO:tasks.rados.rados.0.smithi105.stdout:115: finishing write tid 3 to smithi10536496-14 2024-09-15T03:41:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:115: finishing write tid 4 to smithi10536496-14 2024-09-15T03:41:12.062 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 19 (ObjNum 70 snap 5 seq_num 70) dirty exists 2024-09-15T03:41:12.062 INFO:tasks.rados.rados.0.smithi105.stdout:115: left oid 14 (ObjNum 70 snap 5 seq_num 70) 2024-09-15T03:41:12.102 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:12.101+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:12.119 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:1 3.8s0 3:136df151:::smithi10536496-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016470+0000", 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.100976108999999, 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010868803999999999, 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016470+0000", 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016470+0000", 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016473+0000", 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016566+0000", 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2530999999999999e-05 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016603+0000", 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7576000000000001e-05 2024-09-15T03:41:12.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016610+0000", 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0659999999999998e-06 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016739+0000", 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012916100000000001 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016853+0000", 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113711 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017534+0000", 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068121200000000005 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019644+0000", 2024-09-15T03:41:12.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002109625 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027270+0000", 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076264439999999996 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027339+0000", 2024-09-15T03:41:12.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8957999999999999e-05 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:2 3.8s0 3:136df151:::smithi10536496-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.100823973000001, 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012099814, 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:12.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.99e-07 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016626+0000", 2024-09-15T03:41:12.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016636+0000", 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8439999999999998e-06 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018474+0000", 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018381859999999999 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018568+0000", 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4424000000000003e-05 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019035+0000", 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046713100000000002 2024-09-15T03:41:12.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024240+0000", 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052042240000000003 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028633+0000", 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043929340000000002 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028722+0000", 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9912000000000002e-05 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:3 3.8s0 3:136df151:::smithi10536496-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.10080812, 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016082863999999999, 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:12.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016639+0000", 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016640+0000", 2024-09-15T03:41:12.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2729999999999999e-06 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.113e-06 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016650+0000", 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3489999999999999e-06 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019489+0000", 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028386459999999998 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019567+0000", 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7417999999999996e-05 2024-09-15T03:41:12.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029120+0000", 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095539030000000007 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029205+0000", 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4301000000000005e-05 2024-09-15T03:41:12.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032721+0000", 2024-09-15T03:41:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003516576 2024-09-15T03:41:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.138 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.021861+0000", 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.09558571, 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079237200000000004, 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021861+0000", 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021861+0000", 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021862+0000", 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4000000000000001e-07 2024-09-15T03:41:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021935+0000", 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3300000000000006e-05 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021976+0000", 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1060000000000003e-05 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6490000000000008e-06 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022036+0000", 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2571000000000002e-05 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022045+0000", 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2890000000000003e-06 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029746+0000", 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077008099999999998 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029785+0000", 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8300999999999998e-05 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.024276+0000", 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.093170652, 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01044816, 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024276+0000", 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024276+0000", 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024277+0000", 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2440000000000001e-06 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024364+0000", 2024-09-15T03:41:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6335000000000006e-05 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024406+0000", 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2274999999999998e-05 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024414+0000", 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0120000000000005e-06 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024460+0000", 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5697e-05 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024469+0000", 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2830000000000007e-06 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034693+0000", 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010224482 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034724+0000", 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0831999999999999e-05 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 35/32 ECSubWrite(tid=7, reqid=client.4562.0:35, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.025110+0000", 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.092336985999999, 2024-09-15T03:41:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096341759999999995, 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025110+0000", 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025110+0000", 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025110+0000", 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025129+0000", 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8998e-05 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025281+0000", 2024-09-15T03:41:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015109999999999999 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025291+0000", 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.031e-05 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025338+0000", 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6650000000000002e-05 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025347+0000", 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4590000000000004e-06 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034730+0000", 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093825809999999992 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034744+0000", 2024-09-15T03:41:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4231e-05 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.025466+0000", 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.091980524, 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094195500000000005, 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025466+0000", 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025466+0000", 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025466+0000", 2024-09-15T03:41:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.79e-07 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025543+0000", 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6641e-05 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025590+0000", 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6678000000000003e-05 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025598+0000", 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1440000000000003e-06 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025649+0000", 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0902999999999997e-05 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.025659+0000", 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0165e-05 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034848+0000", 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091888360000000006 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034886+0000", 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7904000000000002e-05 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.026338+0000", 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.091108281, 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085651349999999998, 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026338+0000", 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026338+0000", 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026339+0000", 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4900000000000005e-07 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026376+0000", 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6479000000000001e-05 2024-09-15T03:41:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026411+0000", 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5349999999999999e-05 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026418+0000", 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.43e-06 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026459+0000", 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0960000000000001e-05 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026465+0000", 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6239999999999999e-06 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034890+0000", 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084250209999999996 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034903+0000", 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 35/32 ECSubWrite(tid=114, reqid=client.4562.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027392+0000", 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.090055028, 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074219530000000002, 2024-09-15T03:41:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027392+0000", 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027392+0000", 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027393+0000", 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7699999999999992e-07 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027406+0000", 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.358e-05 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027449+0000", 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3278000000000001e-05 2024-09-15T03:41:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027499+0000", 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9293999999999999e-05 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027567+0000", 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8555999999999999e-05 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027576+0000", 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1409999999999994e-06 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034786+0000", 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072094760000000003 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034814+0000", 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7651e-05 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:40, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028161+0000", 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.089285493, 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092038120000000005, 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028161+0000", 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028161+0000", 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028161+0000", 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8900000000000001e-07 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028531+0000", 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036926299999999999 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028567+0000", 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6390999999999998e-05 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028575+0000", 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7570000000000004e-06 2024-09-15T03:41:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028631+0000", 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5855e-05 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028636+0000", 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7429999999999999e-06 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037277+0000", 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086407500000000009 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037365+0000", 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7863999999999998e-05 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:41, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030266+0000", 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.087181000999999, 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071227590000000002, 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030266+0000", 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030266+0000", 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030266+0000", 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7800000000000001e-07 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030285+0000", 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9165e-05 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030321+0000", 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5434e-05 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030328+0000", 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1439999999999997e-06 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030373+0000", 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4665e-05 2024-09-15T03:41:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030378+0000", 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5300000000000004e-06 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037370+0000", 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006992191 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037388+0000", 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7952e-05 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.036283+0000", 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.081163909000001, 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069778710000000001, 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036283+0000", 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036283+0000", 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036283+0000", 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0200000000000001e-07 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036473+0000", 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000189886 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036503+0000", 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9317999999999999e-05 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036520+0000", 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6830999999999999e-05 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036564+0000", 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4478999999999999e-05 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.036572+0000", 2024-09-15T03:41:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5750000000000003e-06 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043234+0000", 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624229999999998 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043261+0000", 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6557e-05 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.037157+0000", 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.080289378, 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088840310000000006, 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037157+0000", 2024-09-15T03:41:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037157+0000", 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037158+0000", 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6499999999999998e-07 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037178+0000", 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9757000000000001e-05 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037215+0000", 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6828e-05 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037221+0000", 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6769999999999998e-06 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037366+0000", 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000144671 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0999999999999993e-06 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.046014+0000", 2024-09-15T03:41:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086384789999999993 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.046041+0000", 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7654e-05 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 35/32 ECSubWrite(tid=13, reqid=client.4562.0:35, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.037401+0000", 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.080045143, 2024-09-15T03:41:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086867869999999996, 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037401+0000", 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037401+0000", 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037402+0000", 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2600000000000001e-07 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037415+0000", 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2967e-05 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037493+0000", 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7534999999999995e-05 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037503+0000", 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0461e-05 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037564+0000", 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1074000000000005e-05 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.037574+0000", 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.046062+0000", 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084886159999999992 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.046088+0000", 2024-09-15T03:41:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6001000000000001e-05 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:61 3.8s0 3:12b3a6c7:::smithi10536496-21:head [append 0~344064,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043115+0000", 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.074331095, 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010576631, 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 61 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043115+0000", 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043115+0000", 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043117+0000", 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043673+0000", 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055560500000000003 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043688+0000", 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4741e-05 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043694+0000", 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4779999999999996e-06 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1158999999999997e-05 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043831+0000", 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5313e-05 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.044497+0000", 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066600399999999999 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048631+0000", 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041346480000000003 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.053633+0000", 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050020159999999998 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.053692+0000", 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8955000000000001e-05 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:62 3.8s0 3:12b3a6c7:::smithi10536496-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043703+0000", 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.073743164, 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010145628, 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 62 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043703+0000", 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043703+0000", 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043704+0000", 2024-09-15T03:41:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043707+0000", 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7860000000000001e-06 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043712+0000", 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3909999999999997e-06 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043715+0000", 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.644e-06 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045010+0000", 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012955969999999999 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045079+0000", 2024-09-15T03:41:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8565999999999994e-05 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045389+0000", 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030954600000000001 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048659+0000", 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032707980000000001 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.053779+0000", 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051196180000000003 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.053849+0000", 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0073000000000005e-05 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:63 3.8s0 3:12b3a6c7:::smithi10536496-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.073713986, 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012980953, 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 63 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.72e-07 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8279999999999999e-06 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043740+0000", 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9110000000000001e-06 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043742+0000", 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6460000000000002e-06 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045685+0000", 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001942706 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045767+0000", 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2405999999999997e-05 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054190+0000", 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084225670000000006 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054302+0000", 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112184 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.056714+0000", 2024-09-15T03:41:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024113329999999999 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:108 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.026629379999999, 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010882141999999999, 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090818+0000", 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.39e-07 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090819+0000", 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8080000000000001e-06 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090824+0000", 2024-09-15T03:41:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7419999999999997e-06 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090828+0000", 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7890000000000001e-06 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.093335+0000", 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025070370000000002 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.093382+0000", 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7268999999999998e-05 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096751+0000", 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033692040000000002 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0794999999999998e-05 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.101699+0000", 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049071590000000003 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s5 35/32 ECSubWrite(tid=29, reqid=client.4562.0:110, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.096417+0000", 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.021029094999999, 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078332790000000003, 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096417+0000", 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096417+0000", 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096418+0000", 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096431+0000", 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2537e-05 2024-09-15T03:41:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096453+0000", 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2853999999999998e-05 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096464+0000", 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0118e-05 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096504+0000", 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0828999999999998e-05 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096509+0000", 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.95e-06 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104226+0000", 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077162970000000004 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104251+0000", 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5162000000000001e-05 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:138 3.8s0 3:1c642a04:::smithi10536496-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.135164+0000", 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9822830879999991, 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099687830000000002, 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 138 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.135164+0000", 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.135164+0000", 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.135164+0000", 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.135168+0000", 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7919999999999999e-06 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.135175+0000", 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9179999999999997e-06 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.135179+0000", 2024-09-15T03:41:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6580000000000001e-06 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.137203+0000", 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020234440000000001 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.137253+0000", 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9960000000000003e-05 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141411+0000", 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041584170000000002 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141482+0000", 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1097e-05 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145132+0000", 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003650038 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.223 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:12.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:12.225 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:12.223+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:12.325 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:12.324+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:12.326 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:12.327 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:12.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:12.426 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:12.425+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:12.436 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:12.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:12.461 INFO:tasks.rados.rados.0.smithi105.stdout:118: finishing copy_from to smithi10536496-35 2024-09-15T03:41:12.461 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:12.527 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:12.526+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.517492283999999, 2024-09-15T03:41:12.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011155541, 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:12.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029084+0000", 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0603e-05 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029486+0000", 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000402144 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029493+0000", 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.165e-06 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029683+0000", 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000190478 2024-09-15T03:41:12.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029706+0000", 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2849000000000001e-05 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040193+0000", 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010486822999999999 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040228+0000", 2024-09-15T03:41:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4993000000000002e-05 2024-09-15T03:41:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:41, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.516253338, 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098373709999999993, 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030312+0000", 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-15T03:41:12.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030333+0000", 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0635000000000001e-05 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030374+0000", 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030383+0000", 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000004e-06 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030436+0000", 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2605000000000002e-05 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030447+0000", 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1576999999999999e-05 2024-09-15T03:41:12.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040120+0000", 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096721959999999992 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040149+0000", 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9218e-05 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.515668628, 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092836910000000002, 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030897+0000", 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6800000000000009e-07 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030911+0000", 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3589e-05 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030952+0000", 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1619000000000003e-05 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030962+0000", 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4679999999999998e-06 2024-09-15T03:41:12.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031016+0000", 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3671999999999997e-05 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031025+0000", 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6180000000000001e-06 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040164+0000", 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091392020000000008 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040180+0000", 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5554999999999999e-05 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.514942642999999, 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013955543000000001, 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031623+0000", 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031726+0000", 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103491 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031757+0000", 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0908999999999998e-05 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031770+0000", 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2565000000000001e-05 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031863+0000", 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2622999999999997e-05 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031928+0000", 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5984999999999999e-05 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045547+0000", 2024-09-15T03:41:12.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013618655 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045578+0000", 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0565999999999998e-05 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.513979472000001, 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013036834000000001, 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032586+0000", 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032605+0000", 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-05 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032648+0000", 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2763999999999999e-05 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032656+0000", 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032693+0000", 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6701e-05 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032699+0000", 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6849999999999998e-06 2024-09-15T03:41:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045592+0000", 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012893345 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045622+0000", 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0182e-05 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.511342028, 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010405127, 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035224+0000", 2024-09-15T03:41:12.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1800000000000005e-07 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035236+0000", 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1952e-05 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035267+0000", 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035273+0000", 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035320+0000", 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6283999999999999e-05 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035326+0000", 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.105e-06 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045587+0000", 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010261562 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045628+0000", 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0469000000000002e-05 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.507208057, 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090014329999999997, 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039358+0000", 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039502+0000", 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000144272 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039531+0000", 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8753999999999999e-05 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039537+0000", 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2009999999999996e-06 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039596+0000", 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9138000000000002e-05 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039605+0000", 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048329+0000", 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087234679999999998 2024-09-15T03:41:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048358+0000", 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9442999999999999e-05 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=8, reqid=client.4562.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.040877+0000", 2024-09-15T03:41:12.584 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.505687983, 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007439686, 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040877+0000", 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040877+0000", 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040878+0000", 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2440000000000001e-06 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040889+0000", 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0621e-05 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040919+0000", 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0358e-05 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040925+0000", 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0390000000000001e-06 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040972+0000", 2024-09-15T03:41:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7076000000000001e-05 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040984+0000", 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1834000000000001e-05 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048277+0000", 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072927019999999999 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048317+0000", 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9811999999999998e-05 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=7, reqid=client.4562.0:56, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.504994289999999, 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068524500000000004, 2024-09-15T03:41:12.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041571+0000", 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999996e-07 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041600+0000", 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9142999999999999e-05 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041641+0000", 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0720999999999999e-05 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041647+0000", 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8479999999999997e-06 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041686+0000", 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9289000000000003e-05 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.041692+0000", 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1519999999999997e-06 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048373+0000", 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066808750000000002 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048423+0000", 2024-09-15T03:41:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9818000000000003e-05 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=25, reqid=client.4562.0:101, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.462223122999999, 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011334317, 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:12.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084360+0000", 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7618999999999999e-05 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088093+0000", 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037333290000000001 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088098+0000", 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7010000000000004e-06 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088131+0000", 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2653000000000001e-05 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088136+0000", 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4979999999999997e-06 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095632+0000", 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074962550000000003 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095676+0000", 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3516999999999997e-05 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=26, reqid=client.4562.0:103, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.087783+0000", 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.458781481999999, 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082797449999999998, 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087783+0000", 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087783+0000", 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087784+0000", 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8199999999999999e-07 2024-09-15T03:41:12.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088078+0000", 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029388 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088869+0000", 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079139499999999997 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088882+0000", 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2745000000000001e-05 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088931+0000", 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8918000000000002e-05 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088942+0000", 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-15T03:41:12.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096025+0000", 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070827989999999999 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096063+0000", 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8148999999999997e-05 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=27, reqid=client.4562.0:104, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.457728442000001, 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098078109999999996, 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088837+0000", 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088853+0000", 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5770999999999999e-05 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095603+0000", 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067504870000000003 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095612+0000", 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4139999999999994e-06 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095655+0000", 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3417000000000001e-05 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095665+0000", 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0006e-05 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098619+0000", 2024-09-15T03:41:12.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029542119999999999 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098644+0000", 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4975999999999999e-05 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=28, reqid=client.4562.0:109, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.095304+0000", 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.451260681000001, 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088261669999999993, 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095304+0000", 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095304+0000", 2024-09-15T03:41:12.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095305+0000", 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.018e-06 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095583+0000", 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027799500000000001 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096535+0000", 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095136699999999995 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096542+0000", 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.289e-06 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096602+0000", 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9731999999999997e-05 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096614+0000", 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2115e-05 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104103+0000", 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074892919999999998 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104130+0000", 2024-09-15T03:41:12.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7359e-05 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=29, reqid=client.4562.0:110, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.096493+0000", 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.450072027999999, 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076757680000000003, 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096493+0000", 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096493+0000", 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096494+0000", 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4399999999999999e-07 2024-09-15T03:41:12.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096512+0000", 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8726e-05 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099857+0000", 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033441619999999999 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099862+0000", 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5759999999999996e-06 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099907+0000", 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4409999999999997e-05 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099912+0000", 2024-09-15T03:41:12.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0390000000000004e-06 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104144+0000", 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042326580000000003 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104169+0000", 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4352999999999999e-05 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=30, to_read={3:eb414ad4:::smithi10536496-35:head=0,4096,0}, subchunks={3:eb414ad4:::smithi10536496-35:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.446733152, 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071743270000000003, 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099832+0000", 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3300000000000001e-07 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.099840+0000", 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6179999999999997e-06 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104946+0000", 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051057949999999998 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104951+0000", 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.729e-06 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.104990+0000", 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9252999999999998e-05 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.107006+0000", 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020161990000000002 2024-09-15T03:41:12.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=32, to_read={3:edf86ef6:::smithi10536496-37:head=0,4096,0}, subchunks={3:edf86ef6:::smithi10536496-37:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.441124757000001, 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039281828999999997, 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105441+0000", 2024-09-15T03:41:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105447+0000", 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5010000000000001e-06 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141752+0000", 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036305044000000002 2024-09-15T03:41:12.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141762+0000", 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3940000000000002e-06 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141935+0000", 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017359499999999999 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.144722+0000", 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00278671 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=34, reqid=client.4562.0:140, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.404051730000001, 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038895118999999999, 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142514+0000", 2024-09-15T03:41:12.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.449e-05 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180934+0000", 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038406005999999999 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180939+0000", 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4270000000000001e-06 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180984+0000", 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5751000000000003e-05 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180995+0000", 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0332e-05 2024-09-15T03:41:12.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181383+0000", 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038812500000000002 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181408+0000", 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5349000000000001e-05 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=18, reqid=client.4562.0:143, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.145581+0000", 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.400983676999999, 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072727779999999997, 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145581+0000", 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145581+0000", 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145582+0000", 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5499999999999998e-07 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145785+0000", 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020355199999999999 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145809+0000", 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3300000000000001e-05 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145815+0000", 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.494e-06 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145861+0000", 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6054999999999999e-05 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.145879+0000", 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8281999999999999e-05 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.152829+0000", 2024-09-15T03:41:12.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069491350000000004 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.152854+0000", 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5304999999999999e-05 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=39, reqid=client.4562.0:163, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-09-15T03:41:12.635 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.361563918, 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099943480000000001, 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185002+0000", 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185014+0000", 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.205e-05 2024-09-15T03:41:12.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194530+0000", 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095162920000000008 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194535+0000", 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4090000000000002e-06 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194590+0000", 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4234000000000003e-05 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194595+0000", 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4369999999999998e-06 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194977+0000", 2024-09-15T03:41:12.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038248600000000002 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194995+0000", 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799e-05 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=40, reqid=client.4562.0:176, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:12.640 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.352064437999999, 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229890967, 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:12.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194501+0000", 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194514+0000", 2024-09-15T03:41:12.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423889+0000", 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229374775 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423894+0000", 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0669999999999999e-06 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423938+0000", 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3967000000000001e-05 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423948+0000", 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2920000000000001e-06 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424370+0000", 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042234599999999998 2024-09-15T03:41:12.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424391+0000", 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1441999999999999e-05 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:12.645 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:12.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:12.647 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:12.646+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:12.748 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:12.747+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:12.790 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:12.791 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:12.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:12.848 INFO:tasks.rados.rados.0.smithi105.stdout:121: finishing copy_from to smithi10536496-5 2024-09-15T03:41:12.848 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 8 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-15T03:41:12.849 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:12.848+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:12.901 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:12.901 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:12.901 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:12.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:12.949 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:12.948+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:13.012 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.161458510999999, 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013249957, 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 50 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849362+0000", 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849363+0000", 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2099999999999997e-07 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849419+0000", 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6660999999999998e-05 2024-09-15T03:41:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849432+0000", 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2313999999999999e-05 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849437+0000", 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6130000000000003e-06 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854814+0000", 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053769459999999996 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854864+0000", 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9937e-05 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.854960+0000", 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6176999999999998e-05 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860859+0000", 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058984600000000003 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862557+0000", 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016980459999999999 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862590+0000", 2024-09-15T03:41:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3015999999999997e-05 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862612+0000", 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2266e-05 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.161367118000001, 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:41:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.161339012999999, 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:41:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:41:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:41:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:56 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.147452231999999, 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012488551000000001, 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 56 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863368+0000", 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863369+0000", 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0399999999999996e-07 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863435+0000", 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6039999999999998e-05 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863441+0000", 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8250000000000001e-06 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.863444+0000", 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2739999999999999e-06 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867710+0000", 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042658740000000002 2024-09-15T03:41:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867818+0000", 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010801100000000001 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.867963+0000", 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014490799999999999 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.872812+0000", 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048489290000000001 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875774+0000", 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029621589999999998 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875821+0000", 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6966999999999999e-05 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.875857+0000", 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6059999999999997e-05 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.866570898999999, 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012203834, 2024-09-15T03:41:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 97 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144250+0000", 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1699999999999998e-07 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144254+0000", 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8039999999999999e-06 2024-09-15T03:41:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144259+0000", 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3310000000000003e-06 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144262+0000", 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490000000000002e-06 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151586+0000", 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0073231959999999997 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151660+0000", 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4589e-05 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.151733+0000", 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2595000000000005e-05 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154896+0000", 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031635869999999998 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156408+0000", 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001511996 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156435+0000", 2024-09-15T03:41:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6917999999999999e-05 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156454+0000", 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8451999999999999e-05 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.866555578, 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012289876999999999, 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 98 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144265+0000", 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.34e-07 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144267+0000", 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.795e-06 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144270+0000", 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1760000000000002e-06 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144273+0000", 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152322+0000", 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080492889999999994 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152365+0000", 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3634999999999998e-05 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152434+0000", 2024-09-15T03:41:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8738999999999994e-05 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154902+0000", 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024674879999999999 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156492+0000", 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015906309999999999 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156520+0000", 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.811e-05 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156555+0000", 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4585000000000002e-05 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.866545765000001, 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013168571, 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 99 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144275+0000", 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-09-15T03:41:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144313+0000", 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7694000000000002e-05 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144317+0000", 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2719999999999999e-06 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144320+0000", 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152651+0000", 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083311770000000004 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152696+0000", 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4750999999999999e-05 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.152767+0000", 2024-09-15T03:41:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0850000000000001e-05 2024-09-15T03:41:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.156577+0000", 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038109319999999999 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157385+0000", 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080757500000000005 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157410+0000", 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5306000000000001e-05 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.157444+0000", 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3192999999999999e-05 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.866497906999999, 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:41:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:41:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:12 3.3s0 3:cce4e8f8:::smithi10536496-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018097+0000", 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.992723435, 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012025098, 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018097+0000", 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018097+0000", 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018098+0000", 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018102+0000", 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6940000000000001e-06 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018109+0000", 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4749999999999996e-06 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018123+0000", 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4192000000000001e-05 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021889+0000", 2024-09-15T03:41:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037652800000000002 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5224999999999997e-05 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027411+0000", 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054271440000000001 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027497+0000", 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6318999999999998e-05 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030122+0000", 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026250959999999999 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=8, to_read={3:fff44eab:::smithi10536496-10:head=0,4096,0}, subchunks={3:fff44eab:::smithi10536496-10:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.976921562999999, 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046423195, 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033900+0000", 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033906+0000", 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080075+0000", 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046169038000000003 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080080+0000", 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2024-09-15T03:41:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080125+0000", 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5151000000000002e-05 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080322+0000", 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019686500000000001 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:96 3.2s0 3:41f9c41d:::smithi10536496-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.936053978, 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013049477, 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 96 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074767+0000", 2024-09-15T03:41:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074770+0000", 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8890000000000002e-06 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074776+0000", 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.074779+0000", 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3759999999999999e-06 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.077403+0000", 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026236649999999999 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.077475+0000", 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2298999999999993e-05 2024-09-15T03:41:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.085774+0000", 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008299117 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.085861+0000", 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6575000000000004e-05 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.087816+0000", 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019556719999999999 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 35/32 ECSubWriteReply(tid=118, last_complete=35'2, committed=1, applied=1))", 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.923989281000001, 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021508349, 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086832+0000", 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086835+0000", 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0479999999999999e-06 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108275+0000", 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021439848000000001 2024-09-15T03:41:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108283+0000", 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8609999999999998e-06 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108320+0000", 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7330999999999999e-05 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108340+0000", 2024-09-15T03:41:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9531999999999999e-05 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.902415915000001, 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041863826999999999, 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108419+0000", 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149106+0000", 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040686870999999999 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149110+0000", 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.335e-06 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149157+0000", 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7131e-05 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149164+0000", 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150245+0000", 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010804580000000001 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150269+0000", 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4128e-05 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:129 3.3s0 3:c25886c1:::smithi10536496-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.890665017, 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012678227, 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120156+0000", 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0100000000000005e-07 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120160+0000", 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.337e-06 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120167+0000", 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.120171+0000", 2024-09-15T03:41:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4179999999999999e-06 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.122623+0000", 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002451646 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.122711+0000", 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999999e-05 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.127891+0000", 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051798429999999999 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.127977+0000", 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5449000000000001e-05 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.132834+0000", 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048573330000000001 2024-09-15T03:41:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:132 3.3s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.885606537999999, 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015465478, 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 132 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125215+0000", 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125216+0000", 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7349999999999999e-06 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125223+0000", 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125227+0000", 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8840000000000004e-06 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129903+0000", 2024-09-15T03:41:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046767349999999996 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129995+0000", 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1706000000000006e-05 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133187+0000", 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031916510000000002 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133234+0000", 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7862999999999999e-05 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140680+0000", 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074451860000000003 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:133 3.3s0 3:ca2bd2fa:::smithi10536496-45:head [append 0~753664,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.129629+0000", 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.881191275000001, 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011270361, 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 133 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129629+0000", 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129629+0000", 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129630+0000", 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4099999999999997e-07 2024-09-15T03:41:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130352+0000", 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072137799999999995 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130361+0000", 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8629999999999997e-06 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130365+0000", 2024-09-15T03:41:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9650000000000002e-06 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130443+0000", 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8471000000000001e-05 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130576+0000", 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013331199999999999 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.131404+0000", 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00082763200000000002 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.132975+0000", 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015708619999999999 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140821+0000", 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078463049999999996 2024-09-15T03:41:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140900+0000", 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8632000000000002e-05 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:135 3.3s0 3:ca2bd2fa:::smithi10536496-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.130397+0000", 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.880423317, 2024-09-15T03:41:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01259819, 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 135 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130397+0000", 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130397+0000", 2024-09-15T03:41:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130398+0000", 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7800000000000002e-07 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130399+0000", 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130404+0000", 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.318e-06 2024-09-15T03:41:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.130406+0000", 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.481e-06 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133695+0000", 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032885599999999998 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133744+0000", 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9679999999999999e-05 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141649+0000", 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079043429999999994 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141725+0000", 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5959999999999998e-05 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142996+0000", 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001270923 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=25, to_read={3:fc38c57e:::smithi10536496-49:head=0,4096,0}, subchunks={3:fc38c57e:::smithi10536496-49:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.854365776, 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038157982, 2024-09-15T03:41:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156460+0000", 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194316+0000", 2024-09-15T03:41:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037855609999999998 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194320+0000", 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999998e-06 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194361+0000", 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0708999999999999e-05 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194613+0000", 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025227700000000002 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=31, to_read={3:f17f8228:::smithi10536496-40:head=0,167936,0}, subchunks={3:f17f8228:::smithi10536496-40:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.814917586, 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066290004, 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195904+0000", 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195911+0000", 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3030000000000002e-06 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262021+0000", 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066109629000000003 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262025+0000", 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5909999999999998e-06 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262085+0000", 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0183999999999998e-05 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262193+0000", 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107648 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:277 3.13s0 3:ca2bd2fa:::smithi10536496-45:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.981266548, 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0157117019999999, 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 277 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029556+0000", 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.542e-06 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029563+0000", 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2989999999999999e-06 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029577+0000", 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029601+0000", 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3200000000000001e-05 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029661+0000", 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0633e-05 2024-09-15T03:41:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029665+0000", 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5990000000000002e-06 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037509+0000", 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0078444609999999 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037649+0000", 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139476 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.045266+0000", 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076171729999999997 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:13.125 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:13.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T03:41:13.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:41:13.126 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-15T03:41:13.126 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-15T03:41:13.126 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:41:13.127 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:41:13.127 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:13.128 INFO:tasks.rados.rados.0.smithi105.stdout:115: done (7 left) 2024-09-15T03:41:13.128 INFO:tasks.rados.rados.0.smithi105.stdout:116: done (6 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:118: done (5 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:119: done (4 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:120: done (3 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:121: done (2 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:122: done (1 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:123: done (0 left) 2024-09-15T03:41:13.129 INFO:tasks.rados.rados.0.smithi105.stdout:124: snap_remove snap 4 2024-09-15T03:41:13.129 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:13.126+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:13.131 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:13.162 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:41:13.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-15T03:41:13.251 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:13.251 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:13.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:13.264 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:13.263+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:13.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:13.286 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:13.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:13.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:13.329 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:13.328+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:13.346 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:13.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:13.376 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:13.377 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:13.386 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:13.429 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:13.429+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:13.485 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:13.485 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:13.493 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:13.530 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:13.529+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:13.540 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-15T03:41:13.538+0000 7f8f507f1640 -1 mon.a@0(leader).osd e46 definitely_dead 0 2024-09-15T03:41:13.591 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:13.592 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:13.600 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:13.631 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:13.630+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:13.697 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:13.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:13.697 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:13.697 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:13.697 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:13.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:13.732 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:13.731+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.785577088, 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008387205, 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017572+0000", 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1799999999999999e-07 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017597+0000", 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5058999999999998e-05 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017630+0000", 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3037999999999998e-05 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017636+0000", 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.254e-06 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017682+0000", 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6839000000000003e-05 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017691+0000", 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3990000000000004e-06 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025936+0000", 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082456730000000002 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025959+0000", 2024-09-15T03:41:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2725e-05 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=0, reqid=client.4562.0:13, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.783460736, 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011510825000000001, 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019689+0000", 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5799999999999998e-07 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019738+0000", 2024-09-15T03:41:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8652000000000001e-05 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019844+0000", 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106623 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019852+0000", 2024-09-15T03:41:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019952+0000", 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100111 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019961+0000", 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:41:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031174+0000", 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011212744 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031199+0000", 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5278000000000001e-05 2024-09-15T03:41:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.783399138, 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011454499999999999, 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019805+0000", 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5062000000000002e-05 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020203+0000", 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039759399999999997 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020209+0000", 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6189999999999999e-06 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020263+0000", 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4410000000000003e-05 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020283+0000", 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0176000000000001e-05 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031185+0000", 2024-09-15T03:41:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010901569 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031204+0000", 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9663000000000001e-05 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=2, reqid=client.4562.0:14, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.782594660999999, 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010659454, 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020555+0000", 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-07 2024-09-15T03:41:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020569+0000", 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3984e-05 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020610+0000", 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1411e-05 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020616+0000", 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1809999999999998e-06 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022402+0000", 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001785972 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022411+0000", 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6920000000000008e-06 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031203+0000", 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087921389999999992 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031214+0000", 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0902e-05 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021496+0000", 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.781652806, 2024-09-15T03:41:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097223729999999994, 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021496+0000", 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021496+0000", 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021497+0000", 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3499999999999996e-07 2024-09-15T03:41:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021512+0000", 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5534999999999999e-05 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021542+0000", 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9068e-05 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021548+0000", 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9439999999999999e-06 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022379+0000", 2024-09-15T03:41:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083046499999999996 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022388+0000", 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7050000000000005e-06 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031208+0000", 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008820095 2024-09-15T03:41:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031219+0000", 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0926e-05 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.781551279, 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012719667, 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021599+0000", 2024-09-15T03:41:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021655+0000", 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6644999999999997e-05 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021703+0000", 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7549999999999997e-05 2024-09-15T03:41:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021711+0000", 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1519999999999992e-06 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022571+0000", 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086020699999999999 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022577+0000", 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3339999999999998e-06 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034283+0000", 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011705224 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034317+0000", 2024-09-15T03:41:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.485e-05 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.780828849000001, 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012060876, 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022321+0000", 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023195+0000", 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087440700000000001 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023229+0000", 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3905000000000001e-05 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023237+0000", 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.43e-06 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023278+0000", 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1137999999999998e-05 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023286+0000", 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5379999999999997e-06 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034356+0000", 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01107022 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034381+0000", 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5395999999999999e-05 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.779667851999999, 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010863246999999999, 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023482+0000", 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023498+0000", 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6399e-05 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023536+0000", 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7944000000000001e-05 2024-09-15T03:41:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023543+0000", 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023580+0000", 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7379999999999998e-05 2024-09-15T03:41:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023588+0000", 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3950000000000001e-06 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034332+0000", 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010743068 2024-09-15T03:41:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034344+0000", 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2918000000000001e-05 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.779403573, 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010650463000000001, 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023746+0000", 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7700000000000005e-07 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023761+0000", 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5421e-05 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023792+0000", 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0366999999999999e-05 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023797+0000", 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4809999999999997e-06 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025152+0000", 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013547800000000001 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025160+0000", 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.52e-06 2024-09-15T03:41:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034385+0000", 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092252549999999999 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034396+0000", 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1161999999999999e-05 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.778984190999999, 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010190506, 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024165+0000", 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9900000000000001e-07 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024225+0000", 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9503000000000003e-05 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024258+0000", 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2916999999999997e-05 2024-09-15T03:41:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024264+0000", 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8900000000000004e-06 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024305+0000", 2024-09-15T03:41:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1417e-05 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024312+0000", 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0110000000000003e-06 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034347+0000", 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010034405 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034355+0000", 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8640000000000002e-06 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=7, reqid=client.4562.0:35, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024918+0000", 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.778230758999999, 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0094457580000000003, 2024-09-15T03:41:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024918+0000", 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024918+0000", 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024919+0000", 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6899999999999997e-07 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024935+0000", 2024-09-15T03:41:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5875e-05 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024982+0000", 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7382000000000001e-05 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024991+0000", 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6929999999999996e-06 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025476+0000", 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048515 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025483+0000", 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3839999999999999e-06 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034357+0000", 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088736160000000008 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034364+0000", 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9890000000000003e-06 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.777916235999999, 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091838159999999992, 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025233+0000", 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8699999999999999e-07 2024-09-15T03:41:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025404+0000", 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017084199999999999 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025430+0000", 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5649000000000001e-05 2024-09-15T03:41:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025436+0000", 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0610000000000001e-06 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025486+0000", 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0744000000000001e-05 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025495+0000", 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5019999999999992e-06 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034403+0000", 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089080349999999999 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034417+0000", 2024-09-15T03:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3796e-05 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.776816070000001, 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018217534000000001, 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026334+0000", 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8500000000000001e-07 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026371+0000", 2024-09-15T03:41:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6835999999999998e-05 2024-09-15T03:41:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010326484 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036758+0000", 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0989999999999997e-05 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036813+0000", 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050000000000003e-05 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036822+0000", 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3149999999999998e-06 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044526+0000", 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077039439999999999 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.423e-05 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.7s1 35/32 ECSubWrite(tid=2, reqid=client.4562.0:40, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028195+0000", 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.774953655999999, 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089671639999999997, 2024-09-15T03:41:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028195+0000", 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028195+0000", 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028196+0000", 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5599999999999999e-07 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028359+0000", 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016298300000000001 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028390+0000", 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0930999999999999e-05 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028397+0000", 2024-09-15T03:41:13.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0690000000000004e-06 2024-09-15T03:41:13.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028462+0000", 2024-09-15T03:41:13.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028472+0000", 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2739999999999996e-06 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037115+0000", 2024-09-15T03:41:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086439510000000004 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037163+0000", 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7141000000000001e-05 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.as5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.774407181000001, 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083587509999999993, 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028742+0000", 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.98e-07 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028890+0000", 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014750000000000001 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028929+0000", 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9434999999999998e-05 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028937+0000", 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6009999999999997e-06 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029044+0000", 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107129 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029051+0000", 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7060000000000004e-06 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037067+0000", 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080151140000000003 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037101+0000", 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4068e-05 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.as5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.774256499, 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008228997, 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028893+0000", 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000002e-07 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028902+0000", 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.755e-06 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029308+0000", 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000406678 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029314+0000", 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3140000000000003e-06 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031605+0000", 2024-09-15T03:41:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022915790000000002 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031613+0000", 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8469999999999996e-06 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037103+0000", 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054902939999999997 2024-09-15T03:41:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037122+0000", 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8108999999999999e-05 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.036286+0000", 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.76686316, 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083014009999999999, 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036286+0000", 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036286+0000", 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036287+0000", 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036654+0000", 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036737500000000003 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037195+0000", 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054084700000000001 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037201+0000", 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:41:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037245+0000", 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3751000000000002e-05 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037253+0000", 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7700000000000007e-06 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044563+0000", 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073100409999999998 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044587+0000", 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3875000000000001e-05 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.765984543, 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087714329999999993, 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037181+0000", 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5954999999999999e-05 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122876+0000", 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085695108000000006 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122883+0000", 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4069999999999998e-06 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122927+0000", 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552999999999998e-05 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122937+0000", 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.006e-05 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124855+0000", 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001917687 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124879+0000", 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4002000000000002e-05 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 35/32 ECSubWrite(tid=5, reqid=client.4562.0:61, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.044550+0000", 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.758599051999999, 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008671659, 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044550+0000", 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044550+0000", 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1800000000000005e-07 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044834+0000", 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028302200000000002 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044868+0000", 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4654999999999999e-05 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044875+0000", 2024-09-15T03:41:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7499999999999997e-06 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044917+0000", 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1835000000000002e-05 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044926+0000", 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0589999999999992e-06 2024-09-15T03:41:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053178+0000", 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082515959999999999 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053222+0000", 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4023999999999997e-05 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 35/32 ECSubWrite(tid=6, reqid=client.4562.0:62, at_version=35'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.75775848, 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007849824, 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045391+0000", 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7400000000000003e-07 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045403+0000", 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1839e-05 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045428+0000", 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5097999999999999e-05 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045434+0000", 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7490000000000004e-06 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047343+0000", 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001908699 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047351+0000", 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7749999999999998e-06 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053230+0000", 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005879067 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.053240+0000", 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0023e-05 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:13.927 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:13.928 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:13.929 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:13.927+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:13.998 INFO:teuthology.orchestra.run.smithi005.stderr:marked down osd.3. 2024-09-15T03:41:14.000 INFO:tasks.rados.rados.0.smithi105.stdout:124: done (0 left) 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:125: read oid 8 snap -1 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:125: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:126: rollback oid 25 current snap is 7 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 25 to 6 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:127: rmattr oid 50 current snap is 7 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:127: done (2 left) 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 24 current snap is 7 2024-09-15T03:41:14.001 INFO:tasks.rados.rados.0.smithi105.stdout:128: seq_num 72 ranges {0=442368,442368=720896,1163264=49152} 2024-09-15T03:41:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:128: writing smithi10536496-24 from 0 to 442368 tid 1 2024-09-15T03:41:14.008 INFO:tasks.rados.rados.0.smithi105.stdout:128: writing smithi10536496-24 from 442368 to 1163264 tid 2 2024-09-15T03:41:14.009 INFO:tasks.rados.rados.0.smithi105.stdout:128: writing smithi10536496-24 from 1163264 to 1212416 tid 3 2024-09-15T03:41:14.009 INFO:tasks.rados.rados.0.smithi105.stdout:129: read oid 12 snap -1 2024-09-15T03:41:14.009 INFO:tasks.rados.rados.0.smithi105.stdout:129: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T03:41:14.009 INFO:tasks.rados.rados.0.smithi105.stdout:130: setattr oid 4 current snap is 7 2024-09-15T03:41:14.010 INFO:tasks.rados.rados.0.smithi105.stdout:131: snap_create 2024-09-15T03:41:14.022 INFO:tasks.rados.rados.0.smithi105.stdout:126: finishing rollback tid 0 to smithi10536496-25 2024-09-15T03:41:14.023 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-15T03:41:14.023 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-09-15T03:41:14.028 INFO:tasks.rados.rados.0.smithi105.stdout:126: finishing rollback tid 1 to smithi10536496-25 2024-09-15T03:41:14.028 INFO:tasks.rados.rados.0.smithi105.stdout:126: finishing rollback tid 2 to smithi10536496-25 2024-09-15T03:41:14.028 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 19 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-15T03:41:14.036 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:14.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:14.036 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:14.036 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:14.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:14.049 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:41:14.065 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:14.063+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:14.133 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:14.128+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:14.142 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:14.143 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:14.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:14.234 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:14.233+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:14.252 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.230502661999999, 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008773889, 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019951+0000", 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7080000000000002e-05 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020259+0000", 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030882000000000001 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020269+0000", 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4690000000000003e-06 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020353+0000", 2024-09-15T03:41:14.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4121000000000003e-05 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020367+0000", 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.401e-05 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028603+0000", 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082364340000000008 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028667+0000", 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3665999999999997e-05 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.257 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.229655384999999, 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075521720000000002, 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020741+0000", 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020828+0000", 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7108999999999996e-05 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020883+0000", 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4818000000000003e-05 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020888+0000", 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.763e-06 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020921+0000", 2024-09-15T03:41:14.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2780000000000001e-05 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020926+0000", 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8359999999999997e-06 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028240+0000", 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073138810000000004 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028293+0000", 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2281000000000003e-05 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.229563535, 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075072999999999997, 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020833+0000", 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4400000000000001e-07 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020843+0000", 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020899+0000", 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6416000000000003e-05 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020905+0000", 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4670000000000003e-06 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022364+0000", 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001459606 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022373+0000", 2024-09-15T03:41:14.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5229999999999995e-06 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028305+0000", 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059320400000000004 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028340+0000", 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4603000000000001e-05 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022913+0000", 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.227483083999999, 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057677889999999997, 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022913+0000", 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022913+0000", 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022914+0000", 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0410000000000001e-06 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022933+0000", 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9323000000000001e-05 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022975+0000", 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2311000000000002e-05 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022985+0000", 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9739999999999997e-06 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023030+0000", 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4672999999999997e-05 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023039+0000", 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6319999999999997e-06 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028670+0000", 2024-09-15T03:41:14.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056314069999999997 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028681+0000", 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0428000000000001e-05 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:14.271 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.225925246999999, 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072410060000000004, 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.74e-07 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024570+0000", 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9729999999999996e-05 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024599+0000", 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8530999999999998e-05 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024605+0000", 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2280000000000003e-06 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024648+0000", 2024-09-15T03:41:14.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2466000000000003e-05 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024656+0000", 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8690000000000004e-06 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031688+0000", 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070326130000000001 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031712+0000", 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3394999999999999e-05 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=4, reqid=client.4562.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.22498101, 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063223339999999998, 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025416+0000", 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4600000000000004e-07 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025430+0000", 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4636e-05 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025456+0000", 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5718e-05 2024-09-15T03:41:14.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025462+0000", 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025538+0000", 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5916999999999998e-05 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025548+0000", 2024-09-15T03:41:14.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9620000000000005e-06 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031716+0000", 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061672910000000001 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031737+0000", 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1569999999999998e-05 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.22468076, 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061423019999999997, 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025716+0000", 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5199999999999996e-07 2024-09-15T03:41:14.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025913+0000", 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000197252 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025939+0000", 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.552e-05 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025944+0000", 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7320000000000003e-06 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026100+0000", 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575499999999999 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026109+0000", 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3700000000000001e-06 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031831+0000", 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005721271 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031857+0000", 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.675e-05 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026355+0000", 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.224040962, 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005535936, 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026355+0000", 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026355+0000", 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026356+0000", 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026372+0000", 2024-09-15T03:41:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6211000000000001e-05 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026412+0000", 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0034999999999999e-05 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026423+0000", 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0923e-05 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016739529999999999 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028105+0000", 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5569999999999996e-06 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031878+0000", 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00377298 2024-09-15T03:41:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031891+0000", 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2584e-05 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.214024695999999, 2024-09-15T03:41:14.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059101470000000001, 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036372+0000", 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036625+0000", 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025348599999999998 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036653+0000", 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8116e-05 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7190000000000007e-06 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036702+0000", 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0129000000000002e-05 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036711+0000", 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042262+0000", 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005551177 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042281+0000", 2024-09-15T03:41:14.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9004e-05 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.213209809, 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051090550000000004, 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037186+0000", 2024-09-15T03:41:14.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037200+0000", 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3396e-05 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037226+0000", 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6001000000000001e-05 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037231+0000", 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5130000000000004e-06 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037300+0000", 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8978000000000003e-05 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037309+0000", 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6230000000000003e-06 2024-09-15T03:41:14.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042287+0000", 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049783509999999998 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042295+0000", 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.039505+0000", 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.210890785, 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056020430000000001, 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039505+0000", 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039505+0000", 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039506+0000", 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6899999999999996e-07 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039678+0000", 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000172604 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039710+0000", 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1216000000000001e-05 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039716+0000", 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1009999999999997e-06 2024-09-15T03:41:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039760+0000", 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4122000000000002e-05 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039768+0000", 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5290000000000008e-06 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045086+0000", 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053174069999999997 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045107+0000", 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1294999999999999e-05 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/32 ECSubWrite(tid=7, reqid=client.4562.0:56, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:14.304 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.041278+0000", 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.209117675, 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038484719999999999, 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041278+0000", 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041278+0000", 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041279+0000", 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4499999999999997e-07 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041291+0000", 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2024e-05 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041315+0000", 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3745000000000001e-05 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041319+0000", 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0260000000000001e-06 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041350+0000", 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1723000000000001e-05 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041356+0000", 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2549999999999997e-06 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045115+0000", 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003759685 2024-09-15T03:41:14.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045127+0000", 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1369000000000001e-05 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.309 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=16, reqid=client.4562.0:118, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.107844+0000", 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.142551963000001, 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003797749, 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.107844+0000", 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.107844+0000", 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.107845+0000", 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2099999999999995e-07 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108237+0000", 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039230100000000001 2024-09-15T03:41:14.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108268+0000", 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0599000000000003e-05 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108274+0000", 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7460000000000003e-06 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108333+0000", 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8322999999999997e-05 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.108342+0000", 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.111620+0000", 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032782419999999998 2024-09-15T03:41:14.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.111642+0000", 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1348e-05 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:124, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.133746114999999, 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043267160000000004, 2024-09-15T03:41:14.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.116650+0000", 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.99e-07 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117002+0000", 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000351088 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117026+0000", 2024-09-15T03:41:14.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4428999999999999e-05 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117032+0000", 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5940000000000001e-06 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117074+0000", 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2617000000000002e-05 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117084+0000", 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0237e-05 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120955+0000", 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038704239999999999 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120976+0000", 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1628e-05 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:125, at_version=35'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.117220+0000", 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.133175551000001, 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037815679999999999, 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117220+0000", 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117220+0000", 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117221+0000", 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.06e-07 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117237+0000", 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6339999999999999e-05 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117261+0000", 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4043e-05 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117267+0000", 2024-09-15T03:41:14.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.976e-06 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117468+0000", 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020087699999999999 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.117474+0000", 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3520000000000003e-06 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120988+0000", 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003513253 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121002+0000", 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4221e-05 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=60, reqid=client.4562.0:281, at_version=45'25, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:14.323 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.021547+0000", 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.2288484980000001, 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004114876, 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.021547+0000", 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.021547+0000", 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.021548+0000", 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8199999999999999e-07 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.021894+0000", 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034662799999999998 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.021934+0000", 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0268999999999997e-05 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.021954+0000", 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0015000000000001e-05 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.022025+0000", 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0492000000000002e-05 2024-09-15T03:41:14.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.022035+0000", 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0437e-05 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025638+0000", 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036025720000000001 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025662+0000", 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4281e-05 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:14.328 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.2257749860000002, 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067171740000000002, 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024622+0000", 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1489999999999999e-06 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024960+0000", 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000337685 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024999+0000", 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9569e-05 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025010+0000", 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0417e-05 2024-09-15T03:41:14.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025062+0000", 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2043999999999998e-05 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025074+0000", 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2227e-05 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031307+0000", 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062330859999999997 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031338+0000", 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0997000000000002e-05 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.333 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.2243850799999998, 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079502919999999994, 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:14.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026099+0000", 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7769999999999995e-05 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026141+0000", 2024-09-15T03:41:14.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2227000000000001e-05 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026149+0000", 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3730000000000004e-06 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026244+0000", 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5662999999999995e-05 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026255+0000", 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0603e-05 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033934+0000", 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076787699999999997 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033961+0000", 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7206999999999999e-05 2024-09-15T03:41:14.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.2229419250000002, 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065285430000000004, 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027455+0000", 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9299999999999996e-07 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027493+0000", 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7716000000000003e-05 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027532+0000", 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9904000000000003e-05 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027539+0000", 2024-09-15T03:41:14.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027595+0000", 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5841000000000003e-05 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027605+0000", 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7399999999999999e-06 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033967+0000", 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063620969999999997 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033982+0000", 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5389000000000001e-05 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.344 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=64, reqid=client.4562.0:284, at_version=0'0, trim_to=0'0, roll_forward_to=45'28))", 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.2117805709999998, 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090921019999999995, 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038616+0000", 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038629+0000", 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2891000000000001e-05 2024-09-15T03:41:14.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038669+0000", 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9725000000000003e-05 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038682+0000", 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3095000000000001e-05 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038746+0000", 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3850999999999996e-05 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038758+0000", 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2329e-05 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047686+0000", 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089280980000000006 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047707+0000", 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1089000000000001e-05 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:14.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:14.351 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:14.349+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:14.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-15T03:41:14.357+0000 7f8f507f1640 -1 mon.a@0(leader).osd e47 definitely_dead 0 2024-09-15T03:41:14.452 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:14.450+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:14.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:14.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:14.490 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:14.490 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:14.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:14.552 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:14.552+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:14.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:14.596 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:14.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:14.689 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:14.688+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:14.744 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:15 3.0s0 3:092ceaee:::smithi10536496-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.724161414999999, 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021232414000000002, 2024-09-15T03:41:14.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5680000000000001e-06 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018678+0000", 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1760000000000003e-06 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018680+0000", 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.243e-06 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020924+0000", 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022433499999999999 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020978+0000", 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4500000000000003e-05 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032096+0000", 2024-09-15T03:41:14.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011118304000000001 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032182+0000", 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5785000000000006e-05 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039904+0000", 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077221329999999999 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:27 3.0s0 3:0304afdb:::smithi10536496-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.723703017, 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028611145000000001, 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 27 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019131+0000", 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019132+0000", 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7460000000000001e-06 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019139+0000", 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019143+0000", 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6720000000000003e-06 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023415+0000", 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042719689999999996 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023491+0000", 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5638999999999999e-05 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034815+0000", 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011324555 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034893+0000", 2024-09-15T03:41:14.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7682999999999995e-05 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047741+0000", 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01284832 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:14.756 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.720849821, 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:41:14.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.714242574, 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:41:14.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.714224959999999, 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:41:14.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:41:14.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:41:14.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.709499755, 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:64 3.0s0 3:0215fa29:::smithi10536496-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.697311174999999, 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.023494108999999999, 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 64 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045523+0000", 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.358e-06 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045894+0000", 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037004799999999999 2024-09-15T03:41:14.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045904+0000", 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456e-05 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045909+0000", 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6029999999999998e-06 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045938+0000", 2024-09-15T03:41:14.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9054e-05 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046014+0000", 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6334000000000005e-05 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046523+0000", 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000508903 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050383+0000", 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003860086 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068971+0000", 2024-09-15T03:41:14.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018587939000000001 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.069016+0000", 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5327999999999997e-05 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:14.786 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.696918010999999, 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:14.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:41:14.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:41:14.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.696906469, 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:14.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:41:14.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.677714842, 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:14.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:41:14.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.803 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.677230947, 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.677189304000001, 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.674935856999999, 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:41:14.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.673947516, 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:14.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:41:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.617567166000001, 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036705979, 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125267+0000", 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-09-15T03:41:14.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125270+0000", 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161517+0000", 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036246426999999998 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161523+0000", 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161562+0000", 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8571000000000001e-05 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161972+0000", 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041002299999999997 2024-09-15T03:41:14.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.581465067, 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.833 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.577497403000001, 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:41:14.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:287 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 8] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.7145538139999998, 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.820123202, 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 287 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028281+0000", 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5200000000000001e-06 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028286+0000", 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028300+0000", 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3668e-05 2024-09-15T03:41:14.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028307+0000", 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9539999999999998e-06 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029951+0000", 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001644419 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:14.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029955+0000", 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6260000000000001e-06 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037676+0000", 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077205920000000001 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037910+0000", 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234212 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057184+0000", 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019273795999999999 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.059033+0000", 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00184902 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848316+0000", 2024-09-15T03:41:14.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78928352999999996 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848403+0000", 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6583999999999996e-05 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:288 3.10s0 3:092ceaee:::smithi10536496-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:14.844 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.714516873, 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030561617999999999, 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 288 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028317+0000", 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299999999999996e-07 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028320+0000", 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0019999999999998e-06 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028328+0000", 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9379999999999992e-06 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028357+0000", 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8770999999999999e-05 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030005+0000", 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016479140000000001 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030007+0000", 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5229999999999998e-06 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038132+0000", 2024-09-15T03:41:14.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081245950000000001 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038195+0000", 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2838999999999999e-05 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058661+0000", 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020466254999999999 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058771+0000", 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010993999999999999 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058878+0000", 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107268 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 45/44 ECSubWriteReply(tid=39, last_complete=45'16, committed=1, applied=1))", 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.6778824549999998, 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78386887299999997, 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:14.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064956+0000", 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848087+0000", 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78313173400000002 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848099+0000", 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1173e-05 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848179+0000", 2024-09-15T03:41:14.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0921999999999995e-05 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848820+0000", 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064024899999999996 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:14.854 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:14.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:14.857 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:14.855+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:14.957 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:14.956+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s1 3.1f172733 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.95501044099999999, 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.95502673900000001, 2024-09-15T03:41:14.966 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "waiting for peered", 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010866+0000", 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010929+0000", 2024-09-15T03:41:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010944+0000", 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4701e-05 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:14.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010952+0000", 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4610000000000002e-06 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011020+0000", 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8553000000000006e-05 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011023+0000", 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8150000000000002e-06 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:14.970 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:14.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:14.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:14.971 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-09-15T03:41:14.971 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 1 2024-09-15T03:41:14.971 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:14.975 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:15.001 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:41:15.001 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.3. 2024-09-15T03:41:15.002 INFO:tasks.rados.rados.0.smithi105.stdout:126: done (5 left) 2024-09-15T03:41:15.002 INFO:tasks.rados.rados.0.smithi105.stdout:131: done (4 left) 2024-09-15T03:41:15.002 INFO:tasks.rados.rados.0.smithi105.stdout:132: delete oid 26 current snap is 8 2024-09-15T03:41:15.036 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-09-15T03:41:15.036 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:41:15.036 DEBUG:teuthology.orchestra.run.smithi105:> type ceph-objectstore-tool 2024-09-15T03:41:15.053 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-15T03:41:15.053 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-15T03:41:15.057 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:15.057+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:15.085 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:15.086 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:15.086 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:15.086 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:15.086 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:15.094 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:15.159 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:15.158+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:15.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:41:15.184+0000 7f7967acc200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:15.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.200650295000001, 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0090780140000000006, 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0100000000000005e-07 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5333e-05 2024-09-15T03:41:15.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041513+0000", 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084585600000000004 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041517+0000", 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.318e-06 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041556+0000", 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8426000000000001e-05 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041563+0000", 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6930000000000007e-06 2024-09-15T03:41:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042090+0000", 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052617600000000005 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042117+0000", 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6906999999999998e-05 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.186714136999999, 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:41:15.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.175085237999999, 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:15.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:41:15.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:91 3.ds0 3:baae61d4:::smithi10536496-31:head [append 0~507904,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.157865828, 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0073884809999999997, 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.251 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 91 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075824+0000", 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3069999999999999e-06 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076181+0000", 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00035651600000000003 2024-09-15T03:41:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076196+0000", 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5308e-05 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076206+0000", 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4469999999999995e-06 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076297+0000", 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1287999999999997e-05 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076465+0000", 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016844599999999999 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.077478+0000", 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010127140000000001 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080498+0000", 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030201759999999999 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083146+0000", 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026477100000000002 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083212+0000", 2024-09-15T03:41:15.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5568999999999995e-05 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:92 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.15746686, 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078039399999999997, 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.257 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 92 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076223+0000", 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076226+0000", 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6210000000000001e-06 2024-09-15T03:41:15.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076234+0000", 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2740000000000002e-06 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076238+0000", 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4329999999999997e-06 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078113+0000", 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018749769999999999 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078219+0000", 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106016 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078626+0000", 2024-09-15T03:41:15.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040713400000000002 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080519+0000", 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001892712 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083957+0000", 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034382610000000002 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084026+0000", 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8678999999999998e-05 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.157441737999999, 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.263 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:41:15.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:41:15.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.147615989, 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:41:15.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:41:15.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.274 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=14, reqid=client.4562.0:122, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.112925+0000", 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.120763546999999, 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062104780000000002, 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112925+0000", 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112925+0000", 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112926+0000", 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.54e-07 2024-09-15T03:41:15.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112942+0000", 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118742+0000", 2024-09-15T03:41:15.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0057993089999999999 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118746+0000", 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4630000000000003e-06 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118786+0000", 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0299999999999997e-05 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118803+0000", 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6532e-05 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.119119+0000", 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000316142 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.119136+0000", 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6671999999999999e-05 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:124 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~704512,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.114765+0000", 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.118923642, 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0070682460000000003, 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.280 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.281 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 124 2024-09-15T03:41:15.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.281 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.114765+0000", 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.114765+0000", 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.114766+0000", 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2759999999999999e-06 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115094+0000", 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032749199999999997 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115106+0000", 2024-09-15T03:41:15.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2058999999999999e-05 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115111+0000", 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4280000000000004e-06 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4959000000000001e-05 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115311+0000", 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015452599999999999 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116262+0000", 2024-09-15T03:41:15.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095060800000000003 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118414+0000", 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002152798 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121758+0000", 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033438949999999999 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121833+0000", 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5204999999999996e-05 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:125 3.ds0 3:b9f64cc8:::smithi10536496-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.118569064000001, 2024-09-15T03:41:15.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071775770000000001, 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 125 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8000000000000001e-07 2024-09-15T03:41:15.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115122+0000", 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2620000000000001e-06 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115128+0000", 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5729999999999998e-06 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115131+0000", 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6630000000000002e-06 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116937+0000", 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018060249999999999 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116989+0000", 2024-09-15T03:41:15.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1934000000000001e-05 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117215+0000", 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022591199999999999 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118438+0000", 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012237699999999999 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122235+0000", 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037965799999999999 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122297+0000", 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2477999999999994e-05 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.293 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.118538956, 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.294 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.299 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.107622472999999, 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017309390000000001, 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126067+0000", 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126072+0000", 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3920000000000002e-06 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143252+0000", 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017179104000000001 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143257+0000", 2024-09-15T03:41:15.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143315+0000", 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7958000000000003e-05 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143376+0000", 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0535000000000002e-05 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=17, reqid=client.4562.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.143164+0000", 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.090524571, 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060290949999999999, 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143164+0000", 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143164+0000", 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143165+0000", 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999999e-07 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143236+0000", 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0623000000000005e-05 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145546+0000", 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023103920000000001 2024-09-15T03:41:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145554+0000", 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4919999999999996e-06 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145622+0000", 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8671999999999996e-05 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145636+0000", 2024-09-15T03:41:15.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3747999999999999e-05 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.149168+0000", 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035321850000000002 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.149193+0000", 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5215000000000002e-05 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=18, reqid=client.4562.0:143, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.145477+0000", 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.088211885, 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063875959999999997, 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145477+0000", 2024-09-15T03:41:15.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145477+0000", 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145478+0000", 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.141e-06 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.145524+0000", 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6190000000000003e-05 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146492+0000", 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096770799999999996 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146499+0000", 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1749999999999999e-06 2024-09-15T03:41:15.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146544+0000", 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4706999999999997e-05 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146554+0000", 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9709999999999999e-06 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.151836+0000", 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052822040000000004 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.151865+0000", 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8500000000000002e-05 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=19, reqid=client.4562.0:144, at_version=35'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.08723732, 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015149222, 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.85e-07 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146470+0000", 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7309999999999999e-05 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160975+0000", 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014505123 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160981+0000", 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161035+0000", 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3517999999999998e-05 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.065e-05 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161571+0000", 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052504699999999995 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161601+0000", 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=23, reqid=client.4562.0:154, at_version=35'12, trim_to=0'0, roll_forward_to=35'11))", 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.068309922999999, 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.083287300999999994, 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.318 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165380+0000", 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0999999999999997e-07 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165457+0000", 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6823999999999995e-05 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247921+0000", 2024-09-15T03:41:15.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082464198000000002 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247926+0000", 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9880000000000004e-06 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247954+0000", 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8306e-05 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247960+0000", 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7570000000000001e-06 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248650+0000", 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000689816 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248666+0000", 2024-09-15T03:41:15.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6501999999999999e-05 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 39/32 ECSubWrite(tid=29, reqid=osd.3.0:29, at_version=39'17, trim_to=0'0, roll_forward_to=38'15))", 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:06.048706+0000", 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.1849829159999992, 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0059746820000000003, 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048706+0000", 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048706+0000", 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048706+0000", 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3099999999999999e-07 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048729+0000", 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3297000000000001e-05 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048815+0000", 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5538999999999995e-05 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048824+0000", 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8689999999999993e-06 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048879+0000", 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4932999999999997e-05 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.048888+0000", 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6139999999999998e-06 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.054658+0000", 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0057700149999999999 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:06.054681+0000", 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2184e-05 2024-09-15T03:41:15.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=20, reqid=client.4562.0:289, at_version=45'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028907+0000", 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.204781675, 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062185110000000004, 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028907+0000", 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028907+0000", 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028908+0000", 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8700000000000005e-07 2024-09-15T03:41:15.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028936+0000", 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7803000000000001e-05 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028975+0000", 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8775000000000001e-05 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028982+0000", 2024-09-15T03:41:15.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6329999999999996e-06 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029013+0000", 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0825999999999999e-05 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029020+0000", 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035101+0000", 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060813539999999998 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035126+0000", 2024-09-15T03:41:15.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4338999999999999e-05 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=21, reqid=client.4562.0:290, at_version=45'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2034176429999999, 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077833549999999996, 2024-09-15T03:41:15.333 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030272+0000", 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2399999999999997e-07 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030290+0000", 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7683999999999999e-05 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030328+0000", 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8362999999999998e-05 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030335+0000", 2024-09-15T03:41:15.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030374+0000", 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8415999999999999e-05 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030380+0000", 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7649999999999996e-06 2024-09-15T03:41:15.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038032+0000", 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076514269999999997 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038055+0000", 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2776e-05 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=22, reqid=client.4562.0:291, at_version=45'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2027521769999998, 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071344590000000001, 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030955+0000", 2024-09-15T03:41:15.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7567e-05 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.031129+0000", 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000174449 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.031136+0000", 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9480000000000002e-06 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032528+0000", 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001392374 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032540+0000", 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1314e-05 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038058+0000", 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055185049999999999 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038071+0000", 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2838e-05 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.343 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:15.345 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:15.345 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:15.344+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:15.445 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:15.444+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:15.448 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:15.448 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:15.456 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:15.547 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:15.546+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:15.599 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:15.599 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:15.599 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:15.599 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:15.599 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:15.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:15.647 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:15.647+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:15.708 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:18 3.cs0 3:37f43638:::smithi10536496-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.688179295999999, 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016101061999999999, 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.709 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 18 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:15.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018709+0000", 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5000000000000003e-07 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018712+0000", 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9440000000000001e-06 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018720+0000", 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3980000000000002e-06 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.018731+0000", 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1385e-05 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.020559+0000", 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018280589999999999 2024-09-15T03:41:15.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.020635+0000", 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6260000000000005e-05 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031717+0000", 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011081573000000001 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.031804+0000", 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7610999999999999e-05 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.034810+0000", 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030052820000000002 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:30 3.fs0 3:fff44eab:::smithi10536496-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.714 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.687704047, 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062015339000000003, 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 30 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:15.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019185+0000", 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999995e-07 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019188+0000", 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6160000000000002e-06 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019196+0000", 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8699999999999992e-06 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019201+0000", 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:41:15.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019234+0000", 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3402000000000003e-05 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019239+0000", 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1450000000000001e-06 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025625+0000", 2024-09-15T03:41:15.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063864280000000004 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025701+0000", 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6044999999999994e-05 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033568+0000", 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078674130000000002 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033650+0000", 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1404999999999997e-05 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.081199+0000", 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047549438999999999 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.720 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:38 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.024800+0000", 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.682087621000001, 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019995407999999999, 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 38 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024800+0000", 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024800+0000", 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024801+0000", 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000002e-07 2024-09-15T03:41:15.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024806+0000", 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5879999999999999e-06 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024815+0000", 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1390000000000001e-06 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024820+0000", 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025938+0000", 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00111758 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026026+0000", 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8487999999999997e-05 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026351+0000", 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032436800000000001 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026964+0000", 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00061353500000000001 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044742+0000", 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017777830000000001 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044796+0000", 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3792000000000003e-05 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:39 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.024829+0000", 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.682058977000001, 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022622278999999999, 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 39 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024829+0000", 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024829+0000", 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024830+0000", 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9500000000000003e-07 2024-09-15T03:41:15.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024832+0000", 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.977e-06 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024839+0000", 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7129999999999999e-06 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.024843+0000", 2024-09-15T03:41:15.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.76e-06 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026689+0000", 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018464099999999999 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.026768+0000", 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8566000000000006e-05 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044854+0000", 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018086008000000001 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.044950+0000", 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6056000000000003e-05 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047451+0000", 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025012939999999998 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.731 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:53 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.671214635, 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.089580073999999996, 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.732 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 53 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035674+0000", 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7000000000000005e-07 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035677+0000", 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4589999999999998e-06 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035684+0000", 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2659999999999998e-06 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035694+0000", 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8849999999999994e-06 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096889100000000004 2024-09-15T03:41:15.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036718+0000", 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5436000000000003e-05 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037156+0000", 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043800399999999998 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037610+0000", 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00045398200000000002 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125179+0000", 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.087569006000000005 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125253+0000", 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4574999999999997e-05 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:54 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.671188301000001, 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093566412000000002, 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 54 2024-09-15T03:41:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035702+0000", 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1490000000000001e-06 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035709+0000", 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6630000000000004e-06 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035713+0000", 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7780000000000001e-06 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037308+0000", 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015947769999999999 2024-09-15T03:41:15.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037380+0000", 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2293999999999996e-05 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125568+0000", 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.088188213000000001 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125642+0000", 2024-09-15T03:41:15.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4215999999999996e-05 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.129266+0000", 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003623799 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:119 3.fs0 3:f17f8228:::smithi10536496-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:15.743 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.600448288000001, 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046131245000000001, 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 119 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2e-07 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106442+0000", 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0150000000000002e-06 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106446+0000", 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6509999999999999e-06 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106448+0000", 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.21e-06 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107892+0000", 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001444435 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107983+0000", 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0179999999999994e-05 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108344+0000", 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036168399999999997 2024-09-15T03:41:15.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.110942+0000", 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002597495 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152509+0000", 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041567518999999997 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152571+0000", 2024-09-15T03:41:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1636000000000004e-05 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:120 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.600421151999999, 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453923999999997, 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 120 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0400000000000002e-07 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106469+0000", 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.519e-06 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106474+0000", 2024-09-15T03:41:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5090000000000001e-06 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106480+0000", 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.285e-06 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108518+0000", 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020372820000000001 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108590+0000", 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2280999999999995e-05 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152611+0000", 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044021429000000001 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152655+0000", 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3671000000000003e-05 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156921+0000", 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0042656439999999999 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:149 3.fs0 3:fc38c57e:::smithi10536496-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.150214+0000", 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.556673566000001, 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045144404999999999, 2024-09-15T03:41:15.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 149 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150214+0000", 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150214+0000", 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150215+0000", 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1600000000000002e-07 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150217+0000", 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6819999999999999e-06 2024-09-15T03:41:15.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150223+0000", 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0240000000000003e-06 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.150226+0000", 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8750000000000002e-06 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.153821+0000", 2024-09-15T03:41:15.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035946250000000002 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.153869+0000", 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7809999999999998e-05 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156058+0000", 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0021892610000000001 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156134+0000", 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5728000000000004e-05 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039224983999999997 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:171 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.512901926, 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62108922, 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 171 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193987+0000", 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193989+0000", 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6410000000000002e-06 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193994+0000", 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193996+0000", 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00136275 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195404+0000", 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5433999999999999e-05 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195711+0000", 2024-09-15T03:41:15.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030663100000000003 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196156+0000", 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044516899999999998 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815021+0000", 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61886439199999999 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815075+0000", 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4685000000000002e-05 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:172 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.512870613, 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62217043699999997, 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.765 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 172 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194018+0000", 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5699999999999998e-07 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194020+0000", 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194026+0000", 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194028+0000", 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.475e-06 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195860+0000", 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018314609999999999 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195941+0000", 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1464999999999993e-05 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815378+0000", 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61943704499999996 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815470+0000", 2024-09-15T03:41:15.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1351e-05 2024-09-15T03:41:15.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.816188+0000", 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071828500000000004 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:175 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.512824707, 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068787259000000003, 2024-09-15T03:41:15.773 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 175 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194064+0000", 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194065+0000", 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5540000000000001e-06 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194068+0000", 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0730000000000001e-06 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194071+0000", 2024-09-15T03:41:15.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4770000000000002e-06 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195541+0000", 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014705670000000001 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195649+0000", 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108057 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.262851+0000", 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067201307000000002 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 35/32 ECSubWriteReply(tid=30, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.510288384999999, 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61874190500000004, 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196603+0000", 2024-09-15T03:41:15.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7719999999999999e-06 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814918+0000", 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61831525899999995 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814925+0000", 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1629999999999999e-06 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814972+0000", 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7231999999999998e-05 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815342+0000", 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000369019 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.782 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:285 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.024314+0000", 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.6825738239999999, 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015368082999999999, 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 285 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.024314+0000", 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.024314+0000", 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.024315+0000", 2024-09-15T03:41:15.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.024317+0000", 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0779999999999998e-06 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.024324+0000", 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1690000000000003e-06 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.024328+0000", 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3109999999999999e-06 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.027402+0000", 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030741940000000001 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:15.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.027507+0000", 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010413599999999999 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038807+0000", 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011300022 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038894+0000", 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7964e-05 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.039682+0000", 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00078774900000000002 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:300 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.702251215, 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0215413659999999, 2024-09-15T03:41:15.788 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 300 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004638+0000", 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6020000000000001e-06 2024-09-15T03:41:15.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004976+0000", 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033792499999999999 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004994+0000", 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7507e-05 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.005001+0000", 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008081+0000", 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003079675 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008085+0000", 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3420000000000001e-06 2024-09-15T03:41:15.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026131+0000", 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0180457119999999 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026178+0000", 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6953000000000003e-05 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:301 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.697736956, 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0170563379999999, 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 301 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.794 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009153+0000", 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4569999999999999e-06 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009921+0000", 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076796899999999997 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009955+0000", 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3696999999999999e-05 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009971+0000", 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5441e-05 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010025+0000", 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4712000000000001e-05 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010029+0000", 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.63e-06 2024-09-15T03:41:15.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026201+0000", 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016171661 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026207+0000", 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7710000000000001e-06 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:302 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.6969014609999999, 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016248987, 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 302 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7700000000000004e-07 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010036+0000", 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8857999999999999e-05 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010045+0000", 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6200000000000005e-06 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010049+0000", 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3649999999999997e-06 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010226+0000", 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017731600000000001 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010228+0000", 2024-09-15T03:41:15.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3090000000000001e-06 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026231+0000", 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016003078 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026236+0000", 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6639999999999997e-06 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.803 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:303 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.696831105, 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016202901, 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 303 2024-09-15T03:41:15.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1399999999999997e-07 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010061+0000", 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010068+0000", 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9779999999999999e-06 2024-09-15T03:41:15.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010072+0000", 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0489999999999997e-06 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010289+0000", 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021628399999999999 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:15.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010290+0000", 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.201e-06 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026255+0000", 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015965209 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026260+0000", 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6659999999999999e-06 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.808 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:304 3.11s1 3.95fce6f1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.696812252, 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0162086930000001, 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:15.809 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 304 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010078+0000", 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.711e-06 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010082+0000", 2024-09-15T03:41:15.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1320000000000004e-06 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010084+0000", 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5450000000000002e-06 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010311+0000", 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022657600000000001 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010314+0000", 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.565e-06 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026281+0000", 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015966967 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026284+0000", 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:41:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 47/32 ECSubWriteReply(tid=66, last_complete=47'18, committed=1, applied=1))", 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.6942550110000001, 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16187408, 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3799999999999999e-07 2024-09-15T03:41:15.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012636+0000", 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.345e-06 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173941+0000", 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16130529900000001 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173963+0000", 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.243e-05 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174016+0000", 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2948000000000002e-05 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174507+0000", 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049081999999999999 2024-09-15T03:41:15.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.818 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.818 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.818 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:15.818 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:15.818 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:15.819 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:15.820 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:15.818+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:15.919 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:15.919 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:15.919 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:15.919 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:15.920 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:15.919+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:15.927 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:15.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:41:15.945+0000 7f7967acc200 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:41:16.020 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:16.019+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:16.031 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:16.031 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:16.031 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:16.031 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:16.031 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:16.040 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:16.122 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:16.121+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=14, to_read={3:0e027997:::smithi10536496-12:head=0,4096,0}, subchunks={3:0e027997:::smithi10536496-12:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:16.181 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.140737272000001, 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0095256449999999993, 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037696+0000", 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1408e-05 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046920+0000", 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092243440000000006 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046927+0000", 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8090000000000004e-06 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046974+0000", 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6869000000000001e-05 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047209+0000", 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235687 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:49 3.5s0 3:ac8f3d24:::smithi10536496-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.139968947, 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091978560000000008, 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.185 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 49 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038454+0000", 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7099999999999999e-06 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038626+0000", 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017214699999999999 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038636+0000", 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0199e-05 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038641+0000", 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.685e-06 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038679+0000", 2024-09-15T03:41:16.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7731999999999997e-05 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038769+0000", 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0315999999999994e-05 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.039430+0000", 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00066085199999999997 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040743+0000", 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001313346 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047587+0000", 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068434860000000002 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047650+0000", 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3382999999999999e-05 2024-09-15T03:41:16.190 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:50 3.5s0 3:ac8f3d24:::smithi10536496-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.139771898999999, 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091805140000000007, 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.191 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 50 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038650+0000", 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038652+0000", 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.182e-06 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038656+0000", 2024-09-15T03:41:16.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5090000000000004e-06 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038659+0000", 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4030000000000001e-06 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.039949+0000", 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001289884 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040025+0000", 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6005000000000002e-05 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00037365599999999999 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040954+0000", 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055585300000000003 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047765+0000", 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068113210000000004 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047830+0000", 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4237000000000002e-05 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.196 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:51 3.5s0 3:ac8f3d24:::smithi10536496-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.139746871, 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01164283, 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 51 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2599999999999998e-07 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038676+0000", 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.508e-06 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038680+0000", 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.653e-06 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038685+0000", 2024-09-15T03:41:16.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9269999999999996e-06 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040567+0000", 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00188288 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040733+0000", 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016570300000000001 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048135+0000", 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0074020889999999997 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048217+0000", 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2152999999999998e-05 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050317+0000", 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002099591 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=16, reqid=client.4562.0:65, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:16.202 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.130884169, 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020348845000000001, 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5899999999999996e-07 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047553+0000", 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6099999999999998e-05 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066726+0000", 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019172577 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066731+0000", 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3340000000000001e-06 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066780+0000", 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8708999999999997e-05 2024-09-15T03:41:16.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066790+0000", 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9709999999999999e-06 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067856+0000", 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001066259 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067886+0000", 2024-09-15T03:41:16.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9335999999999998e-05 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:67 3.5s0 3:af19ecb4:::smithi10536496-23:head [append 0~376832,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.048294+0000", 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.130126906999999, 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077879739999999996, 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 67 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048294+0000", 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048294+0000", 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048296+0000", 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5570000000000001e-06 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048521+0000", 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022490300000000001 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048533+0000", 2024-09-15T03:41:16.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1965e-05 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048538+0000", 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8459999999999996e-06 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048823+0000", 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028484199999999999 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048934+0000", 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011098400000000001 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.049529+0000", 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00059490699999999999 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051234+0000", 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001705245 2024-09-15T03:41:16.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.055995+0000", 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0047605269999999996 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056082+0000", 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7198000000000001e-05 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:68 3.5s0 3:af19ecb4:::smithi10536496-23:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.048562+0000", 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.129858823999999, 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077016380000000002, 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 68 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048562+0000", 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048562+0000", 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048563+0000", 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1799999999999995e-07 2024-09-15T03:41:16.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048566+0000", 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0630000000000002e-06 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048572+0000", 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5270000000000004e-06 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048577+0000", 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.16e-06 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050389+0000", 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018123900000000001 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050464+0000", 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5113999999999999e-05 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050853+0000", 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038912300000000001 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051342+0000", 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00048895199999999996 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056193+0000", 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0048506410000000002 2024-09-15T03:41:16.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056264+0000", 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1050000000000006e-05 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:69 3.5s0 3:af19ecb4:::smithi10536496-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.129838525, 2024-09-15T03:41:16.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094136830000000008, 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 69 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048583+0000", 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048585+0000", 2024-09-15T03:41:16.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4729999999999999e-06 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048592+0000", 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2360000000000001e-06 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048596+0000", 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.092e-06 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051062+0000", 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024665440000000002 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051144+0000", 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1369e-05 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056556+0000", 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054122759999999997 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056628+0000", 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1699999999999995e-05 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057996+0000", 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001368534 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=18, reqid=client.4562.0:86, at_version=35'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:16.223 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.110998755000001, 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057601141000000002, 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067423+0000", 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2499999999999995e-07 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067437+0000", 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4445e-05 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121493+0000", 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054055923999999998 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121501+0000", 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8949999999999998e-06 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121585+0000", 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3502000000000001e-05 2024-09-15T03:41:16.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121613+0000", 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8503000000000001e-05 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.124988+0000", 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033745590000000001 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125023+0000", 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5787999999999997e-05 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:112 3.5s0 3:a1c61493:::smithi10536496-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.097327+0000", 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.081093716, 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071429179999999998, 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.228 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 112 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097327+0000", 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097327+0000", 2024-09-15T03:41:16.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097329+0000", 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5400000000000001e-06 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097894+0000", 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00056498900000000001 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097918+0000", 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3802999999999999e-05 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097924+0000", 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2929999999999997e-06 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097966+0000", 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2166000000000003e-05 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.098057+0000", 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0667000000000005e-05 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.098698+0000", 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064126299999999999 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099908+0000", 2024-09-15T03:41:16.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001209796 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104409+0000", 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0045013839999999998 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104470+0000", 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1017000000000002e-05 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:113 3.5s0 3:a1c61493:::smithi10536496-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.097936+0000", 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.080484646, 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067138290000000002, 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 113 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.234 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097936+0000", 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097936+0000", 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097937+0000", 2024-09-15T03:41:16.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8800000000000002e-07 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097940+0000", 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0299999999999998e-06 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097947+0000", 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9609999999999999e-06 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097953+0000", 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6919999999999999e-06 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099291+0000", 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001338262 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099379+0000", 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7995e-05 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099771+0000", 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039188800000000001 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.100147+0000", 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00037658800000000002 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104599+0000", 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044514610000000003 2024-09-15T03:41:16.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104650+0000", 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1363999999999998e-05 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:114 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.080462935, 2024-09-15T03:41:16.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093237049999999998, 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 114 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1899999999999998e-07 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097960+0000", 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6130000000000001e-06 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097966+0000", 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3690000000000003e-06 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097970+0000", 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0069999999999999e-06 2024-09-15T03:41:16.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099933+0000", 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019621830000000002 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.100014+0000", 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0852000000000004e-05 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104698+0000", 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046846140000000001 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104783+0000", 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4979000000000001e-05 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107282+0000", 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024986689999999998 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.244 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:117 3.5s0 3:a7103f89:::smithi10536496-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.102000+0000", 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.076421380999999, 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071084190000000004, 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 117 2024-09-15T03:41:16.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.102000+0000", 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.102000+0000", 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.102000+0000", 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9499999999999998e-07 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.102002+0000", 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.102017+0000", 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5265999999999999e-05 2024-09-15T03:41:16.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.102021+0000", 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9740000000000004e-06 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104030+0000", 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002009126 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104111+0000", 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1230999999999995e-05 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107911+0000", 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037999779999999999 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107989+0000", 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7465000000000005e-05 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.109108+0000", 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011194410000000001 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=19, to_read={3:0215fa29:::smithi10536496-22:head=0,4096,0}, subchunks={3:0215fa29:::smithi10536496-22:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.0570012, 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039885337, 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.250 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121442+0000", 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2169999999999999e-05 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160985+0000", 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039542246000000003 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160992+0000", 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2840000000000001e-06 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4248e-05 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161305+0000", 2024-09-15T03:41:16.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025900800000000002 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.9s4 35/32 ECSubWrite(tid=25, reqid=client.4562.0:161, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.014366375, 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075462681000000004, 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9299999999999998e-07 2024-09-15T03:41:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164071+0000", 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5299e-05 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238920+0000", 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074849009999999994 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238927+0000", 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4039999999999998e-06 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238986+0000", 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9086000000000003e-05 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238993+0000", 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3719999999999999e-06 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239496+0000", 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050214699999999999 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239517+0000", 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.177e-05 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:289 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:16.258 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1503635899999995, 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016317213000000001, 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 289 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028059+0000", 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3969999999999999e-06 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028066+0000", 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.627e-06 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028082+0000", 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5406000000000001e-05 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028087+0000", 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4809999999999997e-06 2024-09-15T03:41:16.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028181+0000", 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3881999999999993e-05 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028280+0000", 2024-09-15T03:41:16.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9135999999999994e-05 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028913+0000", 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00063293699999999997 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032481+0000", 2024-09-15T03:41:16.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003567677 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044305+0000", 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011823828 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044375+0000", 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9842e-05 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:290 3.5s0 3:a7103f89:::smithi10536496-39:head [rollback b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1503211960000002, 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016591980999999999, 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 290 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.265 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:16.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028104+0000", 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.501e-06 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028112+0000", 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2379999999999997e-06 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028116+0000", 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3290000000000004e-06 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029591+0000", 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001475004 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029762+0000", 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000170557 2024-09-15T03:41:16.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030258+0000", 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049562300000000005 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035019+0000", 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004761276 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044634+0000", 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096147730000000001 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044692+0000", 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8220000000000002e-05 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:291 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:16.270 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1503000280000002, 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019516250999999998, 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 291 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:16.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028123+0000", 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1239999999999999e-06 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028128+0000", 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5449999999999997e-06 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028132+0000", 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0090000000000001e-06 2024-09-15T03:41:16.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030541+0000", 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024089659999999998 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030601+0000", 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0384000000000003e-05 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030921+0000", 2024-09-15T03:41:16.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031995500000000002 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035034+0000", 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0041124630000000002 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047571+0000", 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012537510999999999 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047637+0000", 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5925000000000003e-05 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=22, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1414288890000002, 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011072657, 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.276 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1899999999999999e-07 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036994+0000", 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047471+0000", 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010477119 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047475+0000", 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3989999999999997e-06 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047509+0000", 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3426000000000002e-05 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.048065+0000", 2024-09-15T03:41:16.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055612399999999996 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=23, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1256580339999998, 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92706636600000003, 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9599999999999999e-07 2024-09-15T03:41:16.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052767+0000", 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8199999999999998e-06 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979592+0000", 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92682499399999996 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979604+0000", 2024-09-15T03:41:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1387e-05 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979726+0000", 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122014 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979829+0000", 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103655 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:16.284 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:16.284 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:16.285 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:16.284+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:16.285 INFO:teuthology.orchestra.run.smithi105.stdout:3.ds1 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.as0 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.9s3 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.3s6 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.19s3 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.13s6 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.14s0 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.12s2 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:3.2s2 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:2.2 2024-09-15T03:41:16.286 INFO:teuthology.orchestra.run.smithi105.stdout:2.0 2024-09-15T03:41:16.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:16.386 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:16.385+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:16.402 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:16.402 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:16.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:16.487 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:16.486+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:16.515 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:16.515 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:16.523 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:16.588 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:16.588+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:16.625 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:16.625 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:16.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:16.689 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:16.688+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:16.736 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:16.736 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:16.736 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:16.741 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 3.2s2 --file /var/log/ceph/exp.3.2s2.3 2024-09-15T03:41:16.744 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:16.830 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:16.828+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:16.886 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:16.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:16.886 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:16.886 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:16.887 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:16.891 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:16.890+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:16.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:16.992 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:16.991+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:17.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:41:17.046 INFO:tasks.rados.rados.0.smithi105.stdout:132: done (4 left) 2024-09-15T03:41:17.046 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 38 current snap is 8 2024-09-15T03:41:17.046 INFO:tasks.rados.rados.0.smithi105.stdout:133: seq_num 74 ranges {671744=475136,1146880=622592,1769472=147456} 2024-09-15T03:41:17.054 INFO:tasks.rados.rados.0.smithi105.stdout:133: writing smithi10536496-38 from 671744 to 1146880 tid 1 2024-09-15T03:41:17.057 INFO:tasks.rados.rados.0.smithi105.stdout:133: writing smithi10536496-38 from 1146880 to 1769472 tid 2 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=4, last_complete=34'2, committed=1, applied=1))", 2024-09-15T03:41:17.059 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.019728282999999, 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049531878000000001, 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:17.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6899999999999999e-07 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036257+0000", 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0109999999999999e-06 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085680+0000", 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049422928999999997 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085688+0000", 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9510000000000006e-06 2024-09-15T03:41:17.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085748+0000", 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0078999999999999e-05 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085787+0000", 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8739000000000003e-05 2024-09-15T03:41:17.064 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:101 3.bs0 3:db73af7f:::smithi10536496-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.973459013999999, 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060658139999999999, 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.065 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 101 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:17.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.067 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082525+0000", 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082528+0000", 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0989999999999999e-06 2024-09-15T03:41:17.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082540+0000", 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2673999999999999e-05 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082544+0000", 2024-09-15T03:41:17.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9559999999999999e-06 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083692+0000", 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011471949999999999 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083836+0000", 2024-09-15T03:41:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014488899999999999 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084281+0000", 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044464500000000002 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085132+0000", 2024-09-15T03:41:17.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00085125900000000004 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143106+0000", 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057973867999999998 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143182+0000", 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5862e-05 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.072 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:102 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.973417403999999, 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100342765, 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.073 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 102 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:17.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082569+0000", 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2279999999999998e-06 2024-09-15T03:41:17.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082577+0000", 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9319999999999996e-06 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082584+0000", 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8009999999999998e-06 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084479+0000", 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018943709999999999 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084576+0000", 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7615000000000006e-05 2024-09-15T03:41:17.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143227+0000", 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058651048999999997 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143302+0000", 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4547000000000003e-05 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182908+0000", 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039606548999999998 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=25, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:17.079 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.97032523, 2024-09-15T03:41:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057539988, 2024-09-15T03:41:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.080 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.98e-07 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085660+0000", 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.902e-06 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142986+0000", 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057326329000000002 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142992+0000", 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143038+0000", 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6576999999999998e-05 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143198+0000", 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015933200000000001 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:111 3.7s0 3:edf86ef6:::smithi10536496-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.961601955999999, 2024-09-15T03:41:17.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050567135999999999, 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 111 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094382+0000", 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094384+0000", 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7489999999999999e-06 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094393+0000", 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2500000000000006e-06 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094397+0000", 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8040000000000001e-06 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002394565 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096872+0000", 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0492000000000001e-05 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105071+0000", 2024-09-15T03:41:17.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081983339999999998 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105164+0000", 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3202999999999995e-05 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.144948+0000", 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039784183000000001 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:140 3.bs0 3:dc608269:::smithi10536496-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.139972+0000", 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.9160106, 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041721282999999998, 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 140 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.090 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139972+0000", 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139972+0000", 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139973+0000", 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139975+0000", 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0169999999999999e-06 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139979+0000", 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8210000000000003e-06 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139981+0000", 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2620000000000001e-06 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141767+0000", 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001785479 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141858+0000", 2024-09-15T03:41:17.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1113999999999995e-05 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142297+0000", 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043900500000000002 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143201+0000", 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00090441899999999995 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.181615+0000", 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038414306000000002 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.181694+0000", 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8233999999999997e-05 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:141 3.bs0 3:dc608269:::smithi10536496-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:17.095 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.915978924000001, 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055455817999999997, 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 141 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6600000000000003e-07 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140006+0000", 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5409999999999999e-06 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140011+0000", 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.843e-06 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140013+0000", 2024-09-15T03:41:17.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.621e-06 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002515181 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142605+0000", 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6328999999999994e-05 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182018+0000", 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039413493000000001 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182116+0000", 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8159000000000007e-05 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.195460+0000", 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013343384999999999 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpReadReply(3.bs0 35/32 ECSubReadReply(tid=35, attrs_read=0))", 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.143734+0000", 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.912248935999999, 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039182792000000001, 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143734+0000", 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143734+0000", 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143735+0000", 2024-09-15T03:41:17.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1699999999999997e-07 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143746+0000", 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0986e-05 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182781+0000", 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039034916000000003 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182786+0000", 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7729999999999996e-06 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182814+0000", 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8034999999999998e-05 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182917+0000", 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102365 2024-09-15T03:41:17.104 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:273 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~802816] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.02906488, 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0316972600000001, 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.105 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026920+0000", 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5629999999999999e-06 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027741+0000", 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082160000000000002 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027764+0000", 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2495000000000001e-05 2024-09-15T03:41:17.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027772+0000", 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027837+0000", 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5062999999999997e-05 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027841+0000", 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043211+0000", 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.015370181 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043389+0000", 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017807500000000001 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044733+0000", 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013438510000000001 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048847+0000", 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041134250000000004 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058556+0000", 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097088929999999997 2024-09-15T03:41:17.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058615+0000", 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9623999999999997e-05 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:274 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~376832] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.0265848049999997, 2024-09-15T03:41:17.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.029738034, 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 274 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029399+0000", 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6099999999999999e-07 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029738+0000", 2024-09-15T03:41:17.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00033941400000000001 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029756+0000", 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524e-05 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029763+0000", 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9609999999999999e-06 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029831+0000", 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8169000000000004e-05 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029833+0000", 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:17.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045818+0000", 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0159847550000001 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045920+0000", 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102085 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.046687+0000", 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00076710099999999996 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049023+0000", 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023356919999999999 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059069+0000", 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010046019999999999 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059136+0000", 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7156999999999994e-05 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:275 3.17s0 3:ec08d906:::smithi10536496-14:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.0261929209999998, 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0294973439999999, 2024-09-15T03:41:17.118 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 275 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029791+0000", 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029807+0000", 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6107e-05 2024-09-15T03:41:17.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029828+0000", 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1475999999999999e-05 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029838+0000", 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9879999999999999e-06 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029885+0000", 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6887e-05 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029887+0000", 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5030000000000001e-06 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047307+0000", 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0174202429999999 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047434+0000", 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012744699999999999 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048033+0000", 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00059865199999999997 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049643+0000", 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016096159999999999 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059241+0000", 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095980579999999996 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059287+0000", 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6726e-05 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:276 3.17s0 3:ec08d906:::smithi10536496-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:17.125 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.0261263600000001, 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0302367509999999, 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 276 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.126 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:17.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029862+0000", 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4259999999999996e-06 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029875+0000", 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3132999999999999e-05 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029880+0000", 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029913+0000", 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3079e-05 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029914+0000", 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.164e-06 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048304+0000", 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.018390516 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048405+0000", 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010111300000000001 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059337+0000", 2024-09-15T03:41:17.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010931523 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059408+0000", 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0962000000000003e-05 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.060093+0000", 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068532499999999997 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.131 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:279 3.17s0 3:eb414ad4:::smithi10536496-35:head [copy-from ver 4] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.011839+0000", 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.0441438420000004, 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050037958, 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 279 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.132 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011839+0000", 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011839+0000", 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011842+0000", 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4499999999999998e-06 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011855+0000", 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3529e-05 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011876+0000", 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0809e-05 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011887+0000", 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0648e-05 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012113+0000", 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00022619599999999999 2024-09-15T03:41:17.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012124+0000", 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0978000000000001e-05 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016030+0000", 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039067399999999997 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016035+0000", 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6539999999999998e-06 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048493+0000", 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032458289000000001 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048644+0000", 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015087999999999999 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.057407+0000", 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087630199999999998 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058748+0000", 2024-09-15T03:41:17.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013411849999999999 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.061839+0000", 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030909940000000001 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.061877+0000", 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7586000000000002e-05 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.139 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:280 3.17s0 3:eb414ad4:::smithi10536496-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.011912+0000", 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.0440709479999999, 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046547195999999999, 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.140 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 280 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011912+0000", 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011912+0000", 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011913+0000", 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1500000000000003e-07 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011916+0000", 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.67e-06 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011927+0000", 2024-09-15T03:41:17.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0108e-05 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011932+0000", 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2979999999999999e-06 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012189+0000", 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00025723199999999998 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:17.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012196+0000", 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5760000000000002e-06 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016057+0000", 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038613499999999999 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016058+0000", 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2500000000000005e-07 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048889+0000", 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032831030999999997 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048942+0000", 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2948999999999997e-05 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058223+0000", 2024-09-15T03:41:17.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0092811779999999993 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058328+0000", 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010461199999999999 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058459+0000", 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013165200000000001 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.146 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.0553044379999998, 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.030189832, 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.147 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000680+0000", 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000683+0000", 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6610000000000002e-06 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000708+0000", 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5165999999999999e-05 2024-09-15T03:41:17.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000714+0000", 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6250000000000004e-06 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005624+0000", 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049095390000000001 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005627+0000", 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-06 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030812+0000", 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025185383 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030868+0000", 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6032000000000002e-05 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.151 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.0552488690000001, 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301912289999999, 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:17.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000735+0000", 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000739+0000", 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0119999999999999e-06 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000747+0000", 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1159999999999999e-06 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000751+0000", 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048937850000000003 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:17.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5700000000000002e-07 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030914+0000", 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252686520000001 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030925+0000", 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1248e-05 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.156 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.0551404679999998, 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301360509999999, 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:17.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000843+0000", 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000856+0000", 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2296999999999999e-05 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000869+0000", 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3045e-05 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000878+0000", 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6749999999999997e-06 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005662+0000", 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047836379999999998 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005663+0000", 2024-09-15T03:41:17.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3300000000000002e-07 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030967+0000", 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025303984 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030978+0000", 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1972e-05 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:297 3.7s0 3:e4711aff:::smithi10536496-25:head [append 0~0] snapc c=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000889+0000", 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.0550939459999999, 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021415285999999999, 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 297 2024-09-15T03:41:17.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000889+0000", 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000889+0000", 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000890+0000", 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7999999999999995e-07 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000893+0000", 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1709999999999998e-06 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000921+0000", 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8024000000000001e-05 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000929+0000", 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1990000000000006e-06 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001017+0000", 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8240999999999998e-05 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001026+0000", 2024-09-15T03:41:17.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0489999999999993e-06 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.009518+0000", 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0084920710000000003 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.009688+0000", 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000169693 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.011128+0000", 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014403969999999999 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015543+0000", 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044146469999999998 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.022253+0000", 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067098840000000002 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.022304+0000", 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1329999999999998e-05 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:298 3.7s0 3:e4711aff:::smithi10536496-25:head [rollback c] snapc c=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000944+0000", 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.0550388430000002, 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027237540000000001, 2024-09-15T03:41:17.168 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 298 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000944+0000", 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000944+0000", 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000945+0000", 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8800000000000002e-07 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000950+0000", 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7379999999999999e-06 2024-09-15T03:41:17.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000959+0000", 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3999999999999992e-06 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000963+0000", 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4710000000000001e-06 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001090+0000", 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012669299999999999 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001096+0000", 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2580000000000001e-06 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012316+0000", 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011219303 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012440+0000", 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012406099999999999 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.013822+0000", 2024-09-15T03:41:17.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001382168 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015672+0000", 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018506040000000001 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028143+0000", 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01247068 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028182+0000", 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8575999999999998e-05 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:299 3.7s0 3:e4711aff:::smithi10536496-25:head [append 0~0] snapc c=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000968+0000", 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.0550147249999999, 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027322619999999999, 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.175 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 299 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000968+0000", 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000968+0000", 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000969+0000", 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.32e-07 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000971+0000", 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2010000000000002e-06 2024-09-15T03:41:17.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000977+0000", 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.601e-06 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001007+0000", 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9858000000000001e-05 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001120+0000", 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112538 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001127+0000", 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1319999999999996e-06 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.014248+0000", 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013121224000000001 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.014345+0000", 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7167e-05 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.014832+0000", 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00048681899999999999 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016737+0000", 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019046130000000001 2024-09-15T03:41:17.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028252+0000", 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01151509 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028291+0000", 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8945000000000001e-05 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.181 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.182 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:17.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:17.184 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:17.182+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:133: writing smithi10536496-38 from 1769472 to 1916928 tid 3 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:134: rollback oid 30 current snap is 8 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 30 to 7 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:129: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 21 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:128: finishing write tid 1 to smithi10536496-24 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:129: done (5 left) 2024-09-15T03:41:17.188 INFO:tasks.rados.rados.0.smithi105.stdout:130: done (4 left) 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 18 current snap is 8 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:135: seq_num 75 ranges {1392640=491520,1884160=770048} 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:135: writing smithi10536496-18 from 1392640 to 1884160 tid 1 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:135: writing smithi10536496-18 from 1884160 to 2654208 tid 2 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:136: rollback oid 39 current snap is 8 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 39 to 6 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:128: finishing write tid 2 to smithi10536496-24 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:128: finishing write tid 3 to smithi10536496-24 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:128: finishing write tid 4 to smithi10536496-24 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:125: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:133: finishing write tid 1 to smithi10536496-38 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:125: done (5 left) 2024-09-15T03:41:17.189 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 37 current snap is 8 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:128: finishing write tid 5 to smithi10536496-24 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 13 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:128: left oid 24 (ObjNum 72 snap 7 seq_num 72) 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:134: finishing rollback tid 0 to smithi10536496-30 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:133: finishing write tid 2 to smithi10536496-38 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:137: seq_num 76 ranges {0=507904,507904=507904,1015808=262144} 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:137: writing smithi10536496-37 from 0 to 507904 tid 1 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:137: writing smithi10536496-37 from 507904 to 1015808 tid 2 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:137: writing smithi10536496-37 from 1015808 to 1277952 tid 3 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:128: done (5 left) 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:138: rollback oid 50 current snap is 8 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 50 to 5 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:133: finishing write tid 3 to smithi10536496-38 2024-09-15T03:41:17.190 INFO:tasks.rados.rados.0.smithi105.stdout:134: finishing rollback tid 1 to smithi10536496-30 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:134: finishing rollback tid 2 to smithi10536496-30 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 17 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:133: finishing write tid 4 to smithi10536496-38 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:133: finishing write tid 5 to smithi10536496-38 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 20 (ObjNum 74 snap 8 seq_num 74) dirty exists 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:133: left oid 38 (ObjNum 74 snap 8 seq_num 74) 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:136: finishing rollback tid 0 to smithi10536496-39 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:135: finishing write tid 1 to smithi10536496-18 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:135: finishing write tid 2 to smithi10536496-18 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:136: finishing rollback tid 1 to smithi10536496-39 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:133: done (5 left) 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:134: done (4 left) 2024-09-15T03:41:17.191 INFO:tasks.rados.rados.0.smithi105.stdout:139: rmattr oid 7 current snap is 8 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:140: copy_from oid 5 from oid 10 current snap is 8 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 14 current snap is 8 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:141: seq_num 77 ranges {1409024=524288,1933312=524288,2457600=245760} 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:141: writing smithi10536496-14 from 1409024 to 1933312 tid 1 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:141: writing smithi10536496-14 from 1933312 to 2457600 tid 2 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:141: writing smithi10536496-14 from 2457600 to 2703360 tid 3 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:142: copy_from oid 41 from oid 48 current snap is 8 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 33 snap 0 seq_num 3869399120) dirty dne 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:138: finishing rollback tid 0 to smithi10536496-50 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:138: finishing rollback tid 1 to smithi10536496-50 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:138: finishing rollback tid 2 to smithi10536496-50 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 16 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-15T03:41:17.192 INFO:tasks.rados.rados.0.smithi105.stdout:140: finishing copy_from racing read to smithi10536496-5 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:138: done (7 left) 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:139: done (6 left) 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 31 current snap is 8 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:143: seq_num 78 ranges {507904=557056,1064960=671744,1736704=98304} 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:143: writing smithi10536496-31 from 507904 to 1064960 tid 1 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:143: writing smithi10536496-31 from 1064960 to 1736704 tid 2 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:143: writing smithi10536496-31 from 1736704 to 1835008 tid 3 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:144: read oid 2 snap -1 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:144: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 50 current snap is 8 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:145: seq_num 79 ranges {835584=573440,1409024=425984,1835008=344064} 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:145: writing smithi10536496-50 from 835584 to 1409024 tid 1 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:145: writing smithi10536496-50 from 1409024 to 1835008 tid 2 2024-09-15T03:41:17.193 INFO:tasks.rados.rados.0.smithi105.stdout:145: writing smithi10536496-50 from 1835008 to 2179072 tid 3 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:146: copy_from oid 36 from oid 15 current snap is 8 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:142: finishing copy_from to smithi10536496-41 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:142: got expected ENOENT (src dne) 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:142: finishing copy_from racing read to smithi10536496-41 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:143: finishing write tid 1 to smithi10536496-31 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:143: finishing write tid 2 to smithi10536496-31 2024-09-15T03:41:17.194 INFO:tasks.rados.rados.0.smithi105.stdout:144: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:143: finishing write tid 3 to smithi10536496-31 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:143: finishing write tid 4 to smithi10536496-31 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:143: finishing write tid 5 to smithi10536496-31 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 12 (ObjNum 78 snap 8 seq_num 78) dirty exists 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:143: left oid 31 (ObjNum 78 snap 8 seq_num 78) 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:142: done (9 left) 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:143: done (8 left) 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:144: done (7 left) 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:147: read oid 20 snap -1 2024-09-15T03:41:17.195 INFO:tasks.rados.rados.0.smithi105.stdout:147: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:41:17.196 INFO:tasks.rados.rados.0.smithi105.stdout:148: read oid 13 snap 5 2024-09-15T03:41:17.196 INFO:tasks.rados.rados.0.smithi105.stdout:148: expect (ObjNum 69 snap 5 seq_num 69) 2024-09-15T03:41:17.196 INFO:tasks.rados.rados.0.smithi105.stdout:149: snap_remove snap 7 2024-09-15T03:41:17.283 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:17.282+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:17.326 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:17.326 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:17.326 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:17.326 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:17.335 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:17.384 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:17.384+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:17.436 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:17.436 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:17.436 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:17.436 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:17.436 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:17.444 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:17.485 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:17.485+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:17.586 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:17.585+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:17.589 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:17.589 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:17.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:17.589 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:17.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.589 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.9s5 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.567562004999999, 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0065251390000000001, 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8999999999999996e-07 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019922+0000", 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017537400000000001 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019959+0000", 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7289999999999997e-05 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019969+0000", 2024-09-15T03:41:17.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020062+0000", 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2602000000000005e-05 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020072+0000", 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048e-05 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026239+0000", 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061667700000000002 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026271+0000", 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2231999999999999e-05 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.594 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.9s5 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.023431+0000", 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.563876565999999, 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028587669999999999, 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023431+0000", 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023431+0000", 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023432+0000", 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.082e-06 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023453+0000", 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0424999999999998e-05 2024-09-15T03:41:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023500+0000", 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7051000000000001e-05 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023511+0000", 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0842000000000001e-05 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023574+0000", 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3046000000000006e-05 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023584+0000", 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9890000000000004e-06 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026276+0000", 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026923680000000001 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026290+0000", 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3964e-05 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.029621+0000", 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.557687107, 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025459609999999998, 2024-09-15T03:41:17.599 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029621+0000", 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029621+0000", 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029622+0000", 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1349999999999999e-06 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029829+0000", 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000207407 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029922+0000", 2024-09-15T03:41:17.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3127000000000005e-05 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029934+0000", 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1165000000000001e-05 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030000+0000", 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6540999999999999e-05 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030010+0000", 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8870000000000004e-06 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.032135+0000", 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021250190000000001 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.032167+0000", 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1680000000000002e-05 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.603 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.5661440239999997, 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003379225, 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0400000000000002e-07 2024-09-15T03:41:17.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022429+0000", 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001264734 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022492+0000", 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2748000000000003e-05 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022499+0000", 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0010000000000004e-06 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022553+0000", 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4358999999999999e-05 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022564+0000", 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0618000000000001e-05 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024511+0000", 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019475479999999999 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024543+0000", 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1912999999999998e-05 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=60, reqid=client.4562.0:281, at_version=45'25, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.021473+0000", 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.5658345499999999, 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022801980000000002, 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021473+0000", 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021473+0000", 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021474+0000", 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2200000000000004e-07 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.022087+0000", 2024-09-15T03:41:17.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00061302600000000002 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.022133+0000", 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6159999999999999e-05 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.022142+0000", 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1439999999999992e-06 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.022211+0000", 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8388999999999999e-05 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.022222+0000", 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1559999999999999e-05 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.023727+0000", 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015045970000000001 2024-09-15T03:41:17.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.023754+0000", 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6699999999999998e-05 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.026458+0000", 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.5608498730000004, 2024-09-15T03:41:17.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026774849999999998, 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026458+0000", 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026458+0000", 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026459+0000", 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026626+0000", 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000167617 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026745+0000", 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011841300000000001 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026753+0000", 2024-09-15T03:41:17.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000001e-06 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026888+0000", 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134313 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026899+0000", 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1124000000000001e-05 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029109+0000", 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022096479999999998 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029136+0000", 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6919000000000001e-05 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.618 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.026631+0000", 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.5606766629999997, 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037504040000000002, 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026631+0000", 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026631+0000", 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026632+0000", 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0900000000000002e-07 2024-09-15T03:41:17.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026675+0000", 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3244999999999997e-05 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028134+0000", 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001459382 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028144+0000", 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028494+0000", 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00035034600000000002 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028505+0000", 2024-09-15T03:41:17.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030354+0000", 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018489999999999999 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030382+0000", 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7605000000000001e-05 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.029234+0000", 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.5580738929999995, 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024709860000000001, 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029234+0000", 2024-09-15T03:41:17.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029234+0000", 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029235+0000", 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0500000000000003e-07 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029253+0000", 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7982000000000002e-05 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030584+0000", 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001331884 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030591+0000", 2024-09-15T03:41:17.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9280000000000004e-06 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030648+0000", 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7009999999999998e-05 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030659+0000", 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0512e-05 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031682+0000", 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001023396 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031705+0000", 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2668999999999999e-05 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.628 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/44 ECSubWrite(tid=21, reqid=client.4562.0:290, at_version=45'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.042173+0000", 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.5451344730000001, 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022705659999999999, 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.042173+0000", 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.042173+0000", 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.042174+0000", 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.02e-07 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.042183+0000", 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8360000000000004e-06 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.042351+0000", 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016804799999999999 2024-09-15T03:41:17.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.042356+0000", 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1679999999999997e-06 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043079+0000", 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00072307100000000004 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043088+0000", 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5089999999999999e-06 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044418+0000", 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013293599999999999 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044444+0000", 2024-09-15T03:41:17.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6472e-05 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.5s1 47/44 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=47'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.002555+0000", 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.5847525019999997, 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006765091, 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.634 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002555+0000", 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002555+0000", 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002556+0000", 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2000000000000001e-07 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002577+0000", 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1115999999999999e-05 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002651+0000", 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3701999999999993e-05 2024-09-15T03:41:17.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002657+0000", 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5390000000000004e-06 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.003242+0000", 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058518600000000002 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.003252+0000", 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9930000000000007e-06 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.009295+0000", 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060427470000000002 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.009320+0000", 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5488000000000001e-05 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.638 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=74, reqid=client.4562.0:297, at_version=47'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.575064201, 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004343876, 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3e-07 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012269+0000", 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5018999999999999e-05 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015244+0000", 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029748069999999999 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015252+0000", 2024-09-15T03:41:17.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015331+0000", 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8817000000000001e-05 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015342+0000", 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0570999999999999e-05 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016564+0000", 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012226870000000001 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016588+0000", 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3274000000000001e-05 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.643 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=76, reqid=client.4562.0:299, at_version=47'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.015200+0000", 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.5721074550000003, 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024177439999999999, 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015200+0000", 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015200+0000", 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015201+0000", 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015225+0000", 2024-09-15T03:41:17.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.393e-05 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015973+0000", 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074839000000000004 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015980+0000", 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016374+0000", 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00039373899999999999 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016384+0000", 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0451e-05 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.017597+0000", 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001212393 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.017618+0000", 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1313000000000001e-05 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=77, reqid=client.4562.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=47'19))", 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.5588705699999998, 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92662449099999999, 2024-09-15T03:41:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028438+0000", 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2099999999999996e-07 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028447+0000", 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8759999999999994e-06 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954390+0000", 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92594294600000004 2024-09-15T03:41:17.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954412+0000", 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2218e-05 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954469+0000", 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7000999999999999e-05 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954479+0000", 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5489999999999995e-06 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955037+0000", 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055842499999999998 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955062+0000", 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4754999999999999e-05 2024-09-15T03:41:17.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:310 3.as0 3.bc9e946a (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e48)", 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.5767194399999998, 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99312189699999998, 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 310 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010590+0000", 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.581e-06 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010594+0000", 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.98e-06 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010608+0000", 2024-09-15T03:41:17.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4251e-05 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010617+0000", 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7819999999999999e-06 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010695+0000", 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8144000000000003e-05 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010699+0000", 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8659999999999999e-06 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003696+0000", 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99299729199999998 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003710+0000", 2024-09-15T03:41:17.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4001e-05 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.5826263229999999, 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048967167, 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.658 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:17.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004683+0000", 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2899999999999999e-06 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004689+0000", 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9229999999999999e-06 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004703+0000", 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004719+0000", 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6095000000000001e-05 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005604+0000", 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00088508100000000002 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005608+0000", 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.247e-06 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030447+0000", 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.024839515 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030545+0000", 2024-09-15T03:41:17.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7453000000000004e-05 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.053649+0000", 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023104197999999999 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.5825757309999999, 2024-09-15T03:41:17.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.049664036, 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:17.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004733+0000", 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0100000000000004e-07 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004737+0000", 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9049999999999999e-06 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004745+0000", 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7540000000000006e-06 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004752+0000", 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9000000000000006e-06 2024-09-15T03:41:17.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005631+0000", 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087836100000000003 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005632+0000", 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.47e-07 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030868+0000", 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252366669999999 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030963+0000", 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4665000000000003e-05 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054396+0000", 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023433236 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:17.669 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.582549776, 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0501537240000001, 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:17.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004759+0000", 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.82e-07 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004762+0000", 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2629999999999999e-06 2024-09-15T03:41:17.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004769+0000", 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0260000000000002e-06 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004780+0000", 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0569e-05 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005657+0000", 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087720500000000004 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005658+0000", 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.159e-06 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031236+0000", 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0255778339999999 2024-09-15T03:41:17.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031320+0000", 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4217999999999999e-05 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054912+0000", 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023591867999999998 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpRead(3.19s0 50/49 ECSubRead(tid=0, to_read={3:9a083222:::smithi10536496-8:head=0,245760,0}, subchunks={3:9a083222:::smithi10536496-8:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.55658066100000003, 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022999359999999998, 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030799+0000", 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2212999999999995e-05 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030809+0000", 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6290000000000005e-06 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.032658+0000", 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018488949999999999 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.033027+0000", 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00036919900000000001 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs1 50/32 ECSubWrite(tid=74, reqid=client.4562.0:320, at_version=50'21, trim_to=0'0, roll_forward_to=47'18))", 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.084148+0000", 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.50315990399999999, 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025941219999999999, 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084148+0000", 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084148+0000", 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084149+0000", 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9800000000000003e-07 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085045+0000", 2024-09-15T03:41:17.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000896311 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085097+0000", 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2163999999999997e-05 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085104+0000", 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.878e-06 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085160+0000", 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5855e-05 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085170+0000", 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0254e-05 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086721+0000", 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001551092 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086742+0000", 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0869999999999998e-05 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es4 50/32 ECSubWrite(tid=43, reqid=client.4562.0:354, at_version=50'19, trim_to=0'0, roll_forward_to=50'16))", 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.162811+0000", 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.424496755, 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.005319921, 2024-09-15T03:41:17.684 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.162811+0000", 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.162811+0000", 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.162812+0000", 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.37e-07 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.163028+0000", 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021615600000000001 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167226+0000", 2024-09-15T03:41:17.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004198022 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167232+0000", 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9399999999999999e-06 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167288+0000", 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6029000000000002e-05 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:17.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167298+0000", 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0186999999999999e-05 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.168108+0000", 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00081004799999999995 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.168131+0000", 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2802e-05 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.688 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.689 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.689 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:17.689 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:17.689 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:17.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:17.691 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:17.690+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:17.791 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:17.791+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:17.832 INFO:teuthology.orchestra.run.smithi105.stderr:Exporting 3.2s2 info 3.2s2( v 35'2 (0'0,35'2] local-lis/les=44/45 n=0 ec=32/32 lis/c=44/44 les/c/f=45/45/0 sis=44) 2024-09-15T03:41:17.834 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:17.834 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:17.834 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:17.834 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:17.836 INFO:teuthology.orchestra.run.smithi105.stderr:Read 2#3:41f9c41d:::smithi10536496-32:head# 2024-09-15T03:41:17.837 INFO:teuthology.orchestra.run.smithi105.stderr:Export successful 2024-09-15T03:41:17.842 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:17.892 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:17.892+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:17.943 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:17.944 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:17.944 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:17.944 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:17.944 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:17.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:17.994 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:17.993+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:149: done (9 left) 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:150: read oid 3 snap -1 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:150: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 46 current snap is 8 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:136: finishing rollback tid 2 to smithi10536496-39 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 24 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:145: finishing write tid 1 to smithi10536496-50 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:145: finishing write tid 2 to smithi10536496-50 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:145: finishing write tid 3 to smithi10536496-50 2024-09-15T03:41:18.019 INFO:tasks.rados.rados.0.smithi105.stdout:145: finishing write tid 4 to smithi10536496-50 2024-09-15T03:41:18.020 INFO:tasks.rados.rados.0.smithi105.stdout:151: seq_num 80 ranges {458752=589824,1048576=573440,1622016=196608} 2024-09-15T03:41:18.020 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:41:18.025 INFO:tasks.rados.rados.0.smithi105.stdout:151: writing smithi10536496-46 from 458752 to 1048576 tid 1 2024-09-15T03:41:18.028 INFO:tasks.rados.rados.0.smithi105.stdout:151: writing smithi10536496-46 from 1048576 to 1622016 tid 2 2024-09-15T03:41:18.030 INFO:tasks.rados.rados.0.smithi105.stdout:151: writing smithi10536496-46 from 1622016 to 1818624 tid 3 2024-09-15T03:41:18.030 INFO:tasks.rados.rados.0.smithi105.stdout:136: done (10 left) 2024-09-15T03:41:18.030 INFO:tasks.rados.rados.0.smithi105.stdout:152: delete oid 30 current snap is 8 2024-09-15T03:41:18.030 INFO:tasks.rados.rados.0.smithi105.stdout:145: finishing write tid 5 to smithi10536496-50 2024-09-15T03:41:18.030 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 20 (ObjNum 79 snap 8 seq_num 79) dirty exists 2024-09-15T03:41:18.030 INFO:tasks.rados.rados.0.smithi105.stdout:145: left oid 50 (ObjNum 79 snap 8 seq_num 79) 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:146: finishing copy_from racing read to smithi10536496-36 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:146: finishing copy_from to smithi10536496-36 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:135: finishing write tid 3 to smithi10536496-18 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:140: finishing copy_from to smithi10536496-5 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:41:18.031 INFO:tasks.rados.rados.0.smithi105.stdout:147: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-15T03:41:18.041 INFO:tasks.rados.rados.0.smithi105.stdout:148: expect (ObjNum 69 snap 5 seq_num 69) 2024-09-15T03:41:18.068 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:18.068 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 34/32 ECSubWrite(tid=0, reqid=client.4562.0:4, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017387+0000", 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.049402677, 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001891144, 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017387+0000", 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017387+0000", 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017388+0000", 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.762e-06 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6598999999999999e-05 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017509+0000", 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3826999999999996e-05 2024-09-15T03:41:18.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017522+0000", 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3577000000000001e-05 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017590+0000", 2024-09-15T03:41:18.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7854000000000005e-05 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017602+0000", 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1707e-05 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019248+0000", 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016461399999999999 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019278+0000", 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9677999999999999e-05 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.048374008, 2024-09-15T03:41:18.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023454370000000001, 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018416+0000", 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5300000000000004e-07 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018457+0000", 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021725800000000001 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018754+0000", 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0282000000000001e-05 2024-09-15T03:41:18.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018806+0000", 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018858+0000", 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1873000000000003e-05 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020727+0000", 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001868897 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020761+0000", 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4165999999999998e-05 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=1, reqid=client.4562.0:8, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.046433145000002, 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021286080000000001, 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020357+0000", 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0139999999999999e-06 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020398+0000", 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1109000000000002e-05 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020456+0000", 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7636999999999997e-05 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020465+0000", 2024-09-15T03:41:18.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6230000000000003e-06 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020522+0000", 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7314e-05 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020532+0000", 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:18.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022453+0000", 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019207950000000001 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022485+0000", 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1915000000000002e-05 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.084 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.045730575, 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017195820000000001, 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021059+0000", 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-07 2024-09-15T03:41:18.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021090+0000", 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0531999999999998e-05 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021617+0000", 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00052774400000000002 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021651+0000", 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3312000000000002e-05 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021685+0000", 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4638999999999998e-05 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021694+0000", 2024-09-15T03:41:18.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.149e-06 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022749+0000", 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010543919999999999 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022778+0000", 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9476e-05 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.089 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028006+0000", 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.038783422000002, 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001802898, 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028006+0000", 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028006+0000", 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028007+0000", 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.28e-06 2024-09-15T03:41:18.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028092+0000", 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4579000000000005e-05 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028243+0000", 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015144600000000001 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028257+0000", 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3343000000000001e-05 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028338+0000", 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1169999999999997e-05 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028359+0000", 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1137000000000001e-05 2024-09-15T03:41:18.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029778+0000", 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014188060000000001 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029809+0000", 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1136999999999997e-05 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.094 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.038691861, 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001736865, 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:18.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028097+0000", 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028098+0000", 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028119+0000", 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0625999999999999e-05 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028357+0000", 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000238545 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028781+0000", 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00042363600000000002 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028878+0000", 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7279000000000002e-05 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028887+0000", 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8259999999999999e-06 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029819+0000", 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00093264499999999996 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029834+0000", 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4816e-05 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.038648631000001, 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020128619999999998, 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.49e-07 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028149+0000", 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4889999999999995e-06 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028822+0000", 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067247100000000001 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028830+0000", 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1429999999999998e-06 2024-09-15T03:41:18.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0082999999999998e-05 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028930+0000", 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0395e-05 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030125+0000", 2024-09-15T03:41:18.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011949440000000001 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030154+0000", 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8187999999999998e-05 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:18.104 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.036498+0000", 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.03029171, 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017902619999999999, 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036498+0000", 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036498+0000", 2024-09-15T03:41:18.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036499+0000", 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.540000000000001e-07 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019873100000000001 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036735+0000", 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7564000000000002e-05 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036742+0000", 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7610000000000002e-06 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036811+0000", 2024-09-15T03:41:18.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9587999999999998e-05 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036821+0000", 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0035e-05 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.038256+0000", 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001435183 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.038288+0000", 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1445999999999997e-05 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.995845260999999, 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084043045999999996, 2024-09-15T03:41:18.109 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070945+0000", 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.124e-06 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070959+0000", 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4212999999999999e-05 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154508+0000", 2024-09-15T03:41:18.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083548836000000001 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154515+0000", 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154566+0000", 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1025e-05 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154578+0000", 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154965+0000", 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038690300000000002 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154987+0000", 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2172999999999999e-05 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.113 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=24, reqid=client.4562.0:100, at_version=35'3, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.983315170999999, 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002110823, 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083476+0000", 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.807e-06 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083643+0000", 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016675300000000001 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083675+0000", 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1962000000000003e-05 2024-09-15T03:41:18.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083684+0000", 2024-09-15T03:41:18.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6930000000000002e-06 2024-09-15T03:41:18.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083746+0000", 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2008000000000007e-05 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083756+0000", 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9450000000000005e-06 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085554+0000", 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017979980000000001 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085585+0000", 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0657000000000001e-05 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=12, reqid=client.4562.0:150, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:18.119 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.912545002, 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035676890000000002, 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154245+0000", 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154487+0000", 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000242308 2024-09-15T03:41:18.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156728+0000", 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022401589999999998 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156736+0000", 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3259999999999996e-06 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156781+0000", 2024-09-15T03:41:18.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4583000000000003e-05 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156789+0000", 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8359999999999998e-06 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157761+0000", 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097187399999999996 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157812+0000", 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0695999999999997e-05 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=13, reqid=client.4562.0:151, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.156461+0000", 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.910328182000001, 2024-09-15T03:41:18.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018052599999999999, 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156461+0000", 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156461+0000", 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156462+0000", 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.54e-07 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156706+0000", 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024380699999999999 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157225+0000", 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00051943900000000001 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157233+0000", 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9310000000000008e-06 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157278+0000", 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5082e-05 2024-09-15T03:41:18.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157287+0000", 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4200000000000007e-06 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.158240+0000", 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00095330000000000002 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.158266+0000", 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6526999999999999e-05 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=14, reqid=client.4562.0:152, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.909601112000001, 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028494564, 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157189+0000", 2024-09-15T03:41:18.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3399999999999998e-07 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157206+0000", 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7207e-05 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185008+0000", 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027801691999999999 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185015+0000", 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185068+0000", 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2333000000000002e-05 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185077+0000", 2024-09-15T03:41:18.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0259999999999996e-06 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185660+0000", 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058343299999999996 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185683+0000", 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714e-05 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.133 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=15, reqid=client.4562.0:164, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.882121343, 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027286849999999998, 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184669+0000", 2024-09-15T03:41:18.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000005e-07 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184986+0000", 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000317027 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015847960000000001 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186578+0000", 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3220000000000004e-06 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186619+0000", 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1186000000000002e-05 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186629+0000", 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:18.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187377+0000", 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074841899999999995 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187397+0000", 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9233999999999999e-05 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=17, reqid=client.4562.0:166, at_version=35'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.879904360999999, 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099130431000000005, 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186886+0000", 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186906+0000", 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0482000000000001e-05 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285393+0000", 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098487125999999994 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285401+0000", 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3780000000000003e-06 2024-09-15T03:41:18.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285446+0000", 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028999999999999e-05 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285453+0000", 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285995+0000", 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00054192900000000002 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.286015+0000", 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0579000000000001e-05 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.0452761759999998, 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030110250000000001, 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8299999999999998e-07 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022295+0000", 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078131100000000003 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022331+0000", 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6103000000000003e-05 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022338+0000", 2024-09-15T03:41:18.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6410000000000002e-06 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022426+0000", 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7366000000000003e-05 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022437+0000", 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.131e-05 2024-09-15T03:41:18.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024502+0000", 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002065271 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024524+0000", 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1739999999999999e-05 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.148 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=56, reqid=client.4562.0:270, at_version=44'15, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.023421+0000", 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.0433687239999996, 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001728329, 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023421+0000", 2024-09-15T03:41:18.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023421+0000", 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023421+0000", 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023481+0000", 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9358000000000003e-05 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023516+0000", 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5210999999999999e-05 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023523+0000", 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4889999999999998e-06 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023574+0000", 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0977000000000003e-05 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.023585+0000", 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0635000000000001e-05 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.025127+0000", 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015421569999999999 2024-09-15T03:41:18.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.025149+0000", 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1810999999999998e-05 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 47/32 ECSubWrite(tid=66, reqid=osd.11.0:66, at_version=47'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.058258253, 2024-09-15T03:41:18.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003545678, 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.29e-07 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008562+0000", 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0871999999999999e-05 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008606+0000", 2024-09-15T03:41:18.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4057999999999997e-05 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008618+0000", 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878999999999999e-05 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008722+0000", 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104236 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008732+0000", 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0142999999999999e-05 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012049+0000", 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033168920000000001 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012077+0000", 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7368999999999998e-05 2024-09-15T03:41:18.157 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 50/32 ECSubWrite(tid=73, reqid=client.4562.0:319, at_version=50'20, trim_to=0'0, roll_forward_to=47'18))", 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.079990+0000", 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.98679884500000004, 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018528609999999999, 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.079990+0000", 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.079990+0000", 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.079991+0000", 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5499999999999999e-07 2024-09-15T03:41:18.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.080469+0000", 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00047772599999999999 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.080502+0000", 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3592e-05 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.080511+0000", 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4889999999999995e-06 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.080575+0000", 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4697999999999996e-05 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.080587+0000", 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1599e-05 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.081826+0000", 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001239471 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.081843+0000", 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6931000000000001e-05 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 50/32 ECSubWrite(tid=74, reqid=client.4562.0:320, at_version=50'21, trim_to=0'0, roll_forward_to=47'18))", 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.084717+0000", 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.98207252499999997, 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021220940000000002, 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084717+0000", 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084717+0000", 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084718+0000", 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5000000000000001e-07 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085070+0000", 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00035266600000000001 2024-09-15T03:41:18.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085107+0000", 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6232000000000001e-05 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085116+0000", 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6379999999999999e-06 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:18.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085165+0000", 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8507000000000002e-05 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085174+0000", 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7869999999999996e-06 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086815+0000", 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001640861 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:18.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086839+0000", 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3553e-05 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:18.167 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:18.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:135: finishing write tid 4 to smithi10536496-18 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 22 (ObjNum 75 snap 8 seq_num 75) dirty exists 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:135: left oid 18 (ObjNum 75 snap 8 seq_num 75) 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:135: done (10 left) 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:140: done (9 left) 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:145: done (8 left) 2024-09-15T03:41:18.170 INFO:tasks.rados.rados.0.smithi105.stdout:146: done (7 left) 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:147: done (6 left) 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:148: done (5 left) 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:152: done (4 left) 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:153: rollback oid 41 current snap is 8 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 41 to 5 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:150: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:151: finishing write tid 1 to smithi10536496-46 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:151: finishing write tid 2 to smithi10536496-46 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:151: finishing write tid 3 to smithi10536496-46 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:151: finishing write tid 4 to smithi10536496-46 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:151: finishing write tid 5 to smithi10536496-46 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 14 (ObjNum 80 snap 8 seq_num 80) dirty exists 2024-09-15T03:41:18.171 INFO:tasks.rados.rados.0.smithi105.stdout:151: left oid 46 (ObjNum 80 snap 8 seq_num 80) 2024-09-15T03:41:18.172 INFO:tasks.rados.rados.0.smithi105.stdout:150: done (4 left) 2024-09-15T03:41:18.172 INFO:tasks.rados.rados.0.smithi105.stdout:151: done (3 left) 2024-09-15T03:41:18.172 INFO:tasks.rados.rados.0.smithi105.stdout:154: snap_create 2024-09-15T03:41:18.172 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:18.168+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:18.248 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:18.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:18.269 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:18.269 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:18.268+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:18.282 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.282 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.289 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:18.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:18.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:18.311 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:18.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:18.331 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:18.331 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:18.350 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:18.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:18.353 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 3.2s2 2024-09-15T03:41:18.369 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:18.369+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:18.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:18.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:18.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:18.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:18.393 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.393 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:18.411 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:18.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:18.430 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:18.430 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:18.451 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:18.451 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:18.471 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:18.472 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:18.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:18.471+0000 7f7d1b7fe640 0 -- 172.21.15.5:0/431560931 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f7d1c195b30 msgr2=0x7f7d1c197f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:18.472 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:18.470+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:18.504 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.504 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:18.572 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:18.571+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:18.655 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.655 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:18.673 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:18.672+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:18.693 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:18.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:18.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:18.765 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.766 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:18.775 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:18.774+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:18.875 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:18.874+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:18.875 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.875 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:18.976 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:18.975+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:18.985 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:18.986 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:18.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:19.029 INFO:tasks.rados.rados.0.smithi105.stdout:154: done (3 left) 2024-09-15T03:41:19.029 INFO:tasks.rados.rados.0.smithi105.stdout:155: read oid 13 snap -1 2024-09-15T03:41:19.029 INFO:tasks.rados.rados.0.smithi105.stdout:155: expect (ObjNum 69 snap 5 seq_num 69) 2024-09-15T03:41:19.030 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 7 current snap is 9 2024-09-15T03:41:19.030 INFO:tasks.rados.rados.0.smithi105.stdout:156: seq_num 82 ranges {0=622592,622592=475136,1097728=294912} 2024-09-15T03:41:19.033 INFO:tasks.rados.rados.0.smithi105.stdout:156: writing smithi10536496-7 from 0 to 622592 tid 1 2024-09-15T03:41:19.037 INFO:tasks.rados.rados.0.smithi105.stdout:156: writing smithi10536496-7 from 622592 to 1097728 tid 2 2024-09-15T03:41:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:156: writing smithi10536496-7 from 1097728 to 1392640 tid 3 2024-09-15T03:41:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:157: snap_create 2024-09-15T03:41:19.043 INFO:tasks.rados.rados.0.smithi105.stdout:153: finishing rollback tid 1 to smithi10536496-41 2024-09-15T03:41:19.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3899997488) dirty dne 2024-09-15T03:41:19.048 INFO:tasks.rados.rados.0.smithi105.stdout:156: finishing write tid 1 to smithi10536496-7 2024-09-15T03:41:19.050 INFO:tasks.rados.rados.0.smithi105.stdout:137: finishing write tid 1 to smithi10536496-37 2024-09-15T03:41:19.050 INFO:tasks.rados.rados.0.smithi105.stdout:155: expect (ObjNum 69 snap 5 seq_num 69) 2024-09-15T03:41:19.076 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:19.076+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:19.115 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.115 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:137: finishing write tid 2 to smithi10536496-37 2024-09-15T03:41:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:137: finishing write tid 3 to smithi10536496-37 2024-09-15T03:41:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:137: finishing write tid 4 to smithi10536496-37 2024-09-15T03:41:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:156: finishing write tid 2 to smithi10536496-7 2024-09-15T03:41:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:137: finishing write tid 5 to smithi10536496-37 2024-09-15T03:41:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 25 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:137: left oid 37 (ObjNum 76 snap 8 seq_num 76) 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:156: finishing write tid 3 to smithi10536496-7 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:141: finishing write tid 1 to smithi10536496-14 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:156: finishing write tid 4 to smithi10536496-7 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:141: finishing write tid 2 to smithi10536496-14 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:141: finishing write tid 3 to smithi10536496-14 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:141: finishing write tid 4 to smithi10536496-14 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:156: finishing write tid 5 to smithi10536496-7 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 14 (ObjNum 82 snap 9 seq_num 82) dirty exists 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:156: left oid 7 (ObjNum 82 snap 9 seq_num 82) 2024-09-15T03:41:19.132 INFO:tasks.rados.rados.0.smithi105.stdout:141: finishing write tid 5 to smithi10536496-14 2024-09-15T03:41:19.133 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 30 (ObjNum 77 snap 8 seq_num 77) dirty exists 2024-09-15T03:41:19.133 INFO:tasks.rados.rados.0.smithi105.stdout:141: left oid 14 (ObjNum 77 snap 8 seq_num 77) 2024-09-15T03:41:19.208 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:19.207+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:19.266 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.267 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:19.279 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:19.278+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:19.379 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:19.378+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:19.417 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.417 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:19.492 INFO:teuthology.orchestra.run.smithi105.stdout: marking collection for removal 2024-09-15T03:41:19.492 INFO:teuthology.orchestra.run.smithi105.stdout:setting '_remove' omap key 2024-09-15T03:41:19.492 INFO:teuthology.orchestra.run.smithi105.stdout:finish_remove_pgs 3.2s2_head removing 3.2s2 2024-09-15T03:41:19.497 INFO:teuthology.orchestra.run.smithi105.stdout:Remove successful 2024-09-15T03:41:19.509 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:19.508+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:19.567 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.567 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:19.581 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:19.580+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:19.682 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:19.681+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:19.717 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.718 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:19.783 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:19.782+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:19.841 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.842 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.851 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:19.883 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:19.883+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:19.950 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:19.950 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:19.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:19.985 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:19.984+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:20.007 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.2s2.3 2024-09-15T03:41:20.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:137: done (5 left) 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:141: done (4 left) 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:153: done (3 left) 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:155: done (2 left) 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:156: done (1 left) 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:157: done (0 left) 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:158: copy_from oid 50 from oid 15 current snap is 10 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 5 current snap is 10 2024-09-15T03:41:20.046 INFO:tasks.rados.rados.0.smithi105.stdout:159: seq_num 83 ranges {442368=638976,1081344=622592,1703936=147456} 2024-09-15T03:41:20.051 INFO:tasks.rados.rados.0.smithi105.stdout:159: writing smithi10536496-5 from 442368 to 1081344 tid 1 2024-09-15T03:41:20.055 INFO:tasks.rados.rados.0.smithi105.stdout:159: writing smithi10536496-5 from 1081344 to 1703936 tid 2 2024-09-15T03:41:20.056 INFO:tasks.rados.rados.0.smithi105.stdout:159: writing smithi10536496-5 from 1703936 to 1851392 tid 3 2024-09-15T03:41:20.057 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 41 current snap is 10 2024-09-15T03:41:20.057 INFO:tasks.rados.rados.0.smithi105.stdout:160: seq_num 84 ranges {0=655360,655360=638976,1294336=131072} 2024-09-15T03:41:20.059 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.059 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.060 INFO:tasks.rados.rados.0.smithi105.stdout:160: writing smithi10536496-41 from 0 to 655360 tid 1 2024-09-15T03:41:20.064 INFO:tasks.rados.rados.0.smithi105.stdout:160: writing smithi10536496-41 from 655360 to 1294336 tid 2 2024-09-15T03:41:20.065 INFO:tasks.rados.rados.0.smithi105.stdout:160: writing smithi10536496-41 from 1294336 to 1425408 tid 3 2024-09-15T03:41:20.065 INFO:tasks.rados.rados.0.smithi105.stdout:161: snap_remove snap 9 2024-09-15T03:41:20.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:20.115 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:20.114+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.167 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:20.186 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:20.186+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:20.287 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:20.287+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:20.313 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.313 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:20.388 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:20.387+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:20.417 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.417 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:20.489 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:20.489+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:20.524 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.524 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.533 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:20.590 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:20.589+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:20.631 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.631 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:20.691 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:20.691+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:20.698 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:20.736 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.736 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:20.830 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:20.829+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:20.830 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:20.852 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:20.852 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:20.872 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:20.873 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:20.873 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:20.884 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:20.884 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:20.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:20.893 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:20.894 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:20.913 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:20.913 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:20.933 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:20.934 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:20.933+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:20.954 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:20.973 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:20.993 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:20.993 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:20.994 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:20.994+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:21.013 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:21.031 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:21.031 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:21.033 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:21.033 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:21.039 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:21.045 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:41:21.046 INFO:tasks.rados.rados.0.smithi105.stdout:161: done (3 left) 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:162: rollback oid 17 current snap is 10 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 17 to 6 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:158: finishing copy_from racing read to smithi10536496-50 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:159: finishing write tid 1 to smithi10536496-5 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:158: finishing copy_from to smithi10536496-50 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:159: finishing write tid 2 to smithi10536496-5 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:159: finishing write tid 3 to smithi10536496-5 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:159: finishing write tid 4 to smithi10536496-5 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:159: finishing write tid 5 to smithi10536496-5 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 27 (ObjNum 83 snap 10 seq_num 83) dirty exists 2024-09-15T03:41:21.047 INFO:tasks.rados.rados.0.smithi105.stdout:159: left oid 5 (ObjNum 83 snap 10 seq_num 83) 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: finishing write tid 1 to smithi10536496-41 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: finishing write tid 2 to smithi10536496-41 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: finishing write tid 3 to smithi10536496-41 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: finishing write tid 4 to smithi10536496-41 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: finishing write tid 5 to smithi10536496-41 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 26 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: left oid 41 (ObjNum 84 snap 10 seq_num 84) 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:158: done (3 left) 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:159: done (2 left) 2024-09-15T03:41:21.048 INFO:tasks.rados.rados.0.smithi105.stdout:160: done (1 left) 2024-09-15T03:41:21.049 INFO:tasks.rados.rados.0.smithi105.stdout:163: snap_create 2024-09-15T03:41:21.060 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:21.061 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:21.066 INFO:tasks.rados.rados.0.smithi105.stdout:162: finishing rollback tid 0 to smithi10536496-17 2024-09-15T03:41:21.068 INFO:tasks.rados.rados.0.smithi105.stdout:162: finishing rollback tid 1 to smithi10536496-17 2024-09-15T03:41:21.068 INFO:tasks.rados.rados.0.smithi105.stdout:162: finishing rollback tid 2 to smithi10536496-17 2024-09-15T03:41:21.068 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 13 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T03:41:21.082 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:21.082 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:21.095 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:21.095+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:21.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:21.102 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:21.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:21.121+0000 7f6039bf2640 0 -- 172.21.15.5:0/1792639874 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f60341956e0 msgr2=0x7f6034197ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:21.159 INFO:teuthology.orchestra.run.smithi105.stdout:get_pg_num_history pg_num_history pg_num_history(e46 pg_nums {1={23=1},2={26=8},3={32=16,44=26}} deleted_pools ) 2024-09-15T03:41:21.159 INFO:teuthology.orchestra.run.smithi105.stdout:Importing pgid 3.2s2 2024-09-15T03:41:21.159 INFO:teuthology.orchestra.run.smithi105.stdout:Write 2#3:41f9c41d:::smithi10536496-32:head# 2024-09-15T03:41:21.159 INFO:teuthology.orchestra.run.smithi105.stdout:snapset 0=[]:{} 2024-09-15T03:41:21.160 INFO:teuthology.orchestra.run.smithi105.stdout:write_pg epoch 45 info 3.2s2( v 35'2 (0'0,35'2] local-lis/les=44/45 n=0 ec=32/32 lis/c=44/44 les/c/f=45/45/0 sis=44) 2024-09-15T03:41:21.160 INFO:teuthology.orchestra.run.smithi105.stdout:Import successful 2024-09-15T03:41:21.178 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:21.178 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:15 3.0s0 3:092ceaee:::smithi10536496-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.157840644, 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021232414000000002, 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5680000000000001e-06 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018678+0000", 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1760000000000003e-06 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018680+0000", 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.243e-06 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020924+0000", 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022433499999999999 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020978+0000", 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4500000000000003e-05 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032096+0000", 2024-09-15T03:41:21.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011118304000000001 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032182+0000", 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5785000000000006e-05 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039904+0000", 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077221329999999999 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:27 3.0s0 3:0304afdb:::smithi10536496-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.157382246000001, 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028611145000000001, 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 27 2024-09-15T03:41:21.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019131+0000", 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019132+0000", 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7460000000000001e-06 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019139+0000", 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019143+0000", 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6720000000000003e-06 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023415+0000", 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042719689999999996 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023491+0000", 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5638999999999999e-05 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034815+0000", 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011324555 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034893+0000", 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7682999999999995e-05 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047741+0000", 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01284832 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.190 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.154529050000001, 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:41:21.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:41:21.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.147921802999999, 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:41:21.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:41:21.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.147904188999998, 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:41:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.143178983999999, 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:41:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:64 3.0s0 3:0215fa29:::smithi10536496-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.130990403999999, 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.023494108999999999, 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 64 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045523+0000", 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.358e-06 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045894+0000", 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037004799999999999 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045904+0000", 2024-09-15T03:41:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456e-05 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045909+0000", 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6029999999999998e-06 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045938+0000", 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9054e-05 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046014+0000", 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6334000000000005e-05 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046523+0000", 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000508903 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050383+0000", 2024-09-15T03:41:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003860086 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068971+0000", 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018587939000000001 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.069016+0000", 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5327999999999997e-05 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.13059724, 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:41:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:41:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.130585698000001, 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:41:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.111394070999999, 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:41:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:41:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.110910176000001, 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:41:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:41:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.110868533000001, 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:41:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:41:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.108615086, 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:41:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:41:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:41:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.107626745000001, 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:41:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:41:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.051246395, 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036705979, 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125267+0000", 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125270+0000", 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161517+0000", 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036246426999999998 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161523+0000", 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:41:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161562+0000", 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8571000000000001e-05 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161972+0000", 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041002299999999997 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.015144295999999, 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:41:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:41:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.011176632000002, 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:287 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 8] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.1482330429999994, 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.820123202, 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 287 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028281+0000", 2024-09-15T03:41:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5200000000000001e-06 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028286+0000", 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028300+0000", 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3668e-05 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028307+0000", 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9539999999999998e-06 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029951+0000", 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001644419 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029955+0000", 2024-09-15T03:41:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6260000000000001e-06 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037676+0000", 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077205920000000001 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037910+0000", 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234212 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057184+0000", 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019273795999999999 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.059033+0000", 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00184902 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848316+0000", 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78928352999999996 2024-09-15T03:41:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848403+0000", 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6583999999999996e-05 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:288 3.10s0 3:092ceaee:::smithi10536496-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.148196102, 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030561617999999999, 2024-09-15T03:41:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 288 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028317+0000", 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299999999999996e-07 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028320+0000", 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0019999999999998e-06 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028328+0000", 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9379999999999992e-06 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028357+0000", 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8770999999999999e-05 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030005+0000", 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016479140000000001 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030007+0000", 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5229999999999998e-06 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038132+0000", 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081245950000000001 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038195+0000", 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2838999999999999e-05 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058661+0000", 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020466254999999999 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058771+0000", 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010993999999999999 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058878+0000", 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107268 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 45/44 ECSubWriteReply(tid=39, last_complete=45'16, committed=1, applied=1))", 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.1115616839999998, 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78386887299999997, 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064956+0000", 2024-09-15T03:41:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848087+0000", 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78313173400000002 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848099+0000", 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1173e-05 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848179+0000", 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0921999999999995e-05 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848820+0000", 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064024899999999996 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:21.285 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:21.287 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:21.287 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:21.286+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:21.385 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:21.385 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:21.385 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:21.386 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:21.388 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:21.387+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:21.393 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:21.489 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:21.489+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:21.490 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:21.491 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:21.491 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:21.491 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:21.491 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:21.499 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:21.590 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:21.589+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.564279079999999, 2024-09-15T03:41:21.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0090780140000000006, 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0100000000000005e-07 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5333e-05 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041513+0000", 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084585600000000004 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041517+0000", 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.318e-06 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041556+0000", 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8426000000000001e-05 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041563+0000", 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6930000000000007e-06 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042090+0000", 2024-09-15T03:41:21.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052617600000000005 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042117+0000", 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6906999999999998e-05 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:21.604 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.550342921999999, 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:41:21.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.608 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.538714023000001, 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:41:21.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:91 3.ds0 3:baae61d4:::smithi10536496-31:head [append 0~507904,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.521494613000002, 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0073884809999999997, 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.614 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 91 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075824+0000", 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3069999999999999e-06 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076181+0000", 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00035651600000000003 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076196+0000", 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5308e-05 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076206+0000", 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4469999999999995e-06 2024-09-15T03:41:21.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076297+0000", 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1287999999999997e-05 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076465+0000", 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016844599999999999 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.077478+0000", 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010127140000000001 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080498+0000", 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030201759999999999 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083146+0000", 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026477100000000002 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083212+0000", 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5568999999999995e-05 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:92 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.521095644999999, 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078039399999999997, 2024-09-15T03:41:21.620 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 92 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076223+0000", 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076226+0000", 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6210000000000001e-06 2024-09-15T03:41:21.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076234+0000", 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2740000000000002e-06 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076238+0000", 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4329999999999997e-06 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078113+0000", 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018749769999999999 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078219+0000", 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106016 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078626+0000", 2024-09-15T03:41:21.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040713400000000002 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080519+0000", 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001892712 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083957+0000", 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034382610000000002 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084026+0000", 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8678999999999998e-05 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.521070522999999, 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:41:21.626 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:41:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.511244774000001, 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.632 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:41:21.635 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:125 3.ds0 3:b9f64cc8:::smithi10536496-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.482197848999999, 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071775770000000001, 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 125 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115120+0000", 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8000000000000001e-07 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115122+0000", 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2620000000000001e-06 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115128+0000", 2024-09-15T03:41:21.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5729999999999998e-06 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115131+0000", 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6630000000000002e-06 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116937+0000", 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018060249999999999 2024-09-15T03:41:21.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.116989+0000", 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1934000000000001e-05 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117215+0000", 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022591199999999999 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.118438+0000", 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012237699999999999 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122235+0000", 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037965799999999999 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122297+0000", 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2477999999999994e-05 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.482167741000001, 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.642 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:41:21.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:21.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:41:21.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:41:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:41:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:41:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:41:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:41:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.651 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:21.651 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:21.651 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.471251257999999, 2024-09-15T03:41:21.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017309390000000001, 2024-09-15T03:41:21.651 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126067+0000", 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126072+0000", 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3920000000000002e-06 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143252+0000", 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017179104000000001 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143257+0000", 2024-09-15T03:41:21.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143315+0000", 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7958000000000003e-05 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143376+0000", 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0535000000000002e-05 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.655 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=19, reqid=client.4562.0:144, at_version=35'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.450866104999999, 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015149222, 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:21.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.85e-07 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146470+0000", 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7309999999999999e-05 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160975+0000", 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014505123 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160981+0000", 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161035+0000", 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3517999999999998e-05 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.065e-05 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161571+0000", 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052504699999999995 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161601+0000", 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:41:21.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=23, reqid=client.4562.0:154, at_version=35'12, trim_to=0'0, roll_forward_to=35'11))", 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.431938708000001, 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.083287300999999994, 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165380+0000", 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0999999999999997e-07 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165457+0000", 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6823999999999995e-05 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247921+0000", 2024-09-15T03:41:21.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082464198000000002 2024-09-15T03:41:21.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247926+0000", 2024-09-15T03:41:21.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9880000000000004e-06 2024-09-15T03:41:21.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247954+0000", 2024-09-15T03:41:21.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8306e-05 2024-09-15T03:41:21.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247960+0000", 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7570000000000001e-06 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248650+0000", 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000689816 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248666+0000", 2024-09-15T03:41:21.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6501999999999999e-05 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=21, reqid=client.4562.0:290, at_version=45'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5670464279999994, 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077833549999999996, 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030272+0000", 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2399999999999997e-07 2024-09-15T03:41:21.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030290+0000", 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7683999999999999e-05 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030328+0000", 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8362999999999998e-05 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030335+0000", 2024-09-15T03:41:21.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030374+0000", 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8415999999999999e-05 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030380+0000", 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7649999999999996e-06 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038032+0000", 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076514269999999997 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038055+0000", 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2776e-05 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=22, reqid=client.4562.0:291, at_version=45'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:21.671 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5663809620000002, 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071344590000000001, 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030937+0000", 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030955+0000", 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7567e-05 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.031129+0000", 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000174449 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.031136+0000", 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9480000000000002e-06 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032528+0000", 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001392374 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032540+0000", 2024-09-15T03:41:21.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1314e-05 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038058+0000", 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055185049999999999 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038071+0000", 2024-09-15T03:41:21.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2838e-05 2024-09-15T03:41:21.677 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.677 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.677 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.677 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s1 3.1f172733 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:21.678 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:21.678 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 7.5864536170000001, 2024-09-15T03:41:21.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017692614, 2024-09-15T03:41:21.678 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.678 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.678 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.679 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.679 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.679 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:41:21.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.679 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010866+0000", 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010929+0000", 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010944+0000", 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4701e-05 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010952+0000", 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4610000000000002e-06 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011020+0000", 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8553000000000006e-05 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011023+0000", 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8150000000000002e-06 2024-09-15T03:41:21.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028536+0000", 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017512913 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028557+0000", 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0716e-05 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:310 3.as1 3:5629793d:::smithi10536496-26:head [delete] snapc e=[e,c,b] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.5918584979999997, 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0408854540000001, 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 310 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.684 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005461+0000", 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.643e-06 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005465+0000", 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005480+0000", 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5407999999999999e-05 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005487+0000", 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4229999999999999e-06 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008383+0000", 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002895309 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:21.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008386+0000", 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8949999999999998e-06 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.031743+0000", 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0233571130000001 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.032058+0000", 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000315123 2024-09-15T03:41:21.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.033215+0000", 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011571579999999999 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.037843+0000", 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046277380000000002 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:21.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.045957+0000", 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081145560000000002 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046345+0000", 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000387229 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 50/49 ECSubWrite(tid=151, reqid=client.4562.0:335, at_version=50'20, trim_to=0'0, roll_forward_to=50'18))", 2024-09-15T03:41:21.690 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.466546439, 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067600538000000002, 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130773+0000", 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130954+0000", 2024-09-15T03:41:21.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018089000000000001 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196934+0000", 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065979530999999994 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196943+0000", 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197012+0000", 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8745999999999995e-05 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197025+0000", 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2909e-05 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198352+0000", 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001327188 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198372+0000", 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9933999999999999e-05 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:347 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.4490466419999999, 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089938310000000007, 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:21.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 347 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.58e-07 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148275+0000", 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0269999999999997e-06 2024-09-15T03:41:21.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148282+0000", 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.195e-06 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148286+0000", 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3669999999999999e-06 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.152643+0000", 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043572790000000004 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.152776+0000", 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000133156 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.153340+0000", 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00056305400000000001 2024-09-15T03:41:21.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154283+0000", 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00094319800000000002 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.156961+0000", 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026778100000000001 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157265+0000", 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030428699999999998 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:348 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.4490270389999997, 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010390460000000001, 2024-09-15T03:41:21.701 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 348 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.84e-07 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148293+0000", 2024-09-15T03:41:21.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.858e-06 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148299+0000", 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9780000000000002e-06 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148303+0000", 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8249999999999998e-06 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154925+0000", 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066227459999999997 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.155002+0000", 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6196e-05 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157677+0000", 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026754650000000001 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157794+0000", 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117167 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.158681+0000", 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088684099999999997 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.706 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.18s5 51/49 ECSubWrite(tid=153, reqid=client.4562.0:369, at_version=51'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.5622554559999999, 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007414889, 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035063+0000", 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035480+0000", 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041686700000000002 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035525+0000", 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5948999999999997e-05 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035532+0000", 2024-09-15T03:41:21.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1349999999999997e-06 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035566+0000", 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4785e-05 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035574+0000", 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.042460+0000", 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068862660000000003 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.042477+0000", 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:21.711 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:21.713 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:21.714 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /var/log/ceph/exp.3.2s2.3 2024-09-15T03:41:21.714 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:21.712+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:21.778 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:41:21.821 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:21.821+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:21.851 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:21.852 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:21.852 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:21.852 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:21.860 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:21.914 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:21.914+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:21.998 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:21.998 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:21.998 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:21.998 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:21.998 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:22.006 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:22.016 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:22.015+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:22.045 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:22.061 INFO:tasks.rados.rados.0.smithi105.stdout:162: done (1 left) 2024-09-15T03:41:22.061 INFO:tasks.rados.rados.0.smithi105.stdout:163: done (0 left) 2024-09-15T03:41:22.061 INFO:tasks.rados.rados.0.smithi105.stdout:164: snap_remove snap 6 2024-09-15T03:41:22.074 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:41:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":55,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:41:22.047672+0000","last_up_change":"2024-09-15T03:41:13.994626+0000","last_in_change":"2024-09-15T03:41:14.996464+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:40:57.987730+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"44","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":46,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":33113},{"type":"v1","addr":"172.21.15.105:6803","nonce":33113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":33113},{"type":"v1","addr":"172.21.15.105:6805","nonce":33113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":33113},{"type":"v1","addr":"172.21.15.105:6809","nonce":33113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":33113},{"type":"v1","addr":"172.21.15.105:6807","nonce":33113}]},"public_addr":"172.21.15.105:6803/33113","cluster_addr":"172.21.15.105:6805/33113","heartbeat_back_addr":"172.21.15.105:6809/33113","heartbeat_front_addr":"172.21.15.105:6807/33113","state":["exists"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.532883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.696957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.621678+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-15T03:40:39.668299+0000","dead_epoch":47},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.560569+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.593272+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.525540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.677822+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.597079+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.594754+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.639694+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.691925+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.673878+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.655432+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.605739+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:40:39.574974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3,"length":1},{"begin":6,"length":1},{"begin":8,"length":1},{"begin":10,"length":1},{"begin":14,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:41:22.087 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:40:42.070557+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:40:44.856082+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 29, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-15T03:40:57.987730+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '44', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}] 2024-09-15T03:41:22.088 DEBUG:teuthology.orchestra.run.smithi105:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool .mgr 2024-09-15T03:41:22.106 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:22.106 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:22.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:30 3.fs0 3:fff44eab:::smithi10536496-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.085527666000001, 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062015339000000003, 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.107 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 30 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019185+0000", 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999995e-07 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019188+0000", 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6160000000000002e-06 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019196+0000", 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8699999999999992e-06 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019201+0000", 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019234+0000", 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3402000000000003e-05 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019239+0000", 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1450000000000001e-06 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025625+0000", 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063864280000000004 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025701+0000", 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6044999999999994e-05 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033568+0000", 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078674130000000002 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033650+0000", 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1404999999999997e-05 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.081199+0000", 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047549438999999999 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:53 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.069038253999999, 2024-09-15T03:41:22.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.089580073999999996, 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 53 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035674+0000", 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7000000000000005e-07 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035677+0000", 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4589999999999998e-06 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035684+0000", 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2659999999999998e-06 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035694+0000", 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8849999999999994e-06 2024-09-15T03:41:22.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096889100000000004 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036718+0000", 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5436000000000003e-05 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037156+0000", 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043800399999999998 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037610+0000", 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00045398200000000002 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125179+0000", 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.087569006000000005 2024-09-15T03:41:22.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125253+0000", 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4574999999999997e-05 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:54 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.069011920000001, 2024-09-15T03:41:22.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093566412000000002, 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 54 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035702+0000", 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1490000000000001e-06 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035709+0000", 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6630000000000004e-06 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035713+0000", 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7780000000000001e-06 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037308+0000", 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015947769999999999 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037380+0000", 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2293999999999996e-05 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125568+0000", 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.088188213000000001 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125642+0000", 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4215999999999996e-05 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.129266+0000", 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003623799 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:119 3.fs0 3:f17f8228:::smithi10536496-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.998271906999999, 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046131245000000001, 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.125 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 119 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2e-07 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106442+0000", 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0150000000000002e-06 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106446+0000", 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6509999999999999e-06 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106448+0000", 2024-09-15T03:41:22.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.21e-06 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107892+0000", 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001444435 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107983+0000", 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0179999999999994e-05 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108344+0000", 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036168399999999997 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.110942+0000", 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002597495 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152509+0000", 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041567518999999997 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152571+0000", 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1636000000000004e-05 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:120 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.998244771, 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453923999999997, 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.133 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 120 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:22.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0400000000000002e-07 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106469+0000", 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.519e-06 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106474+0000", 2024-09-15T03:41:22.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5090000000000001e-06 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106480+0000", 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.285e-06 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108518+0000", 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020372820000000001 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108590+0000", 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2280999999999995e-05 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152611+0000", 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044021429000000001 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152655+0000", 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3671000000000003e-05 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156921+0000", 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0042656439999999999 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:171 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:22.138 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.910725544999998, 2024-09-15T03:41:22.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62108922, 2024-09-15T03:41:22.139 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.139 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.139 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.139 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.139 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.140 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 171 2024-09-15T03:41:22.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.140 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193987+0000", 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193989+0000", 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6410000000000002e-06 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193994+0000", 2024-09-15T03:41:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193996+0000", 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:41:22.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00136275 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195404+0000", 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5433999999999999e-05 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195711+0000", 2024-09-15T03:41:22.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030663100000000003 2024-09-15T03:41:22.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196156+0000", 2024-09-15T03:41:22.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044516899999999998 2024-09-15T03:41:22.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815021+0000", 2024-09-15T03:41:22.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61886439199999999 2024-09-15T03:41:22.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815075+0000", 2024-09-15T03:41:22.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4685000000000002e-05 2024-09-15T03:41:22.147 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.147 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.148 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.148 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:172 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.148 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:22.148 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.910694232000001, 2024-09-15T03:41:22.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62217043699999997, 2024-09-15T03:41:22.149 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.149 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.149 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.149 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.149 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.150 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 172 2024-09-15T03:41:22.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.150 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194018+0000", 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5699999999999998e-07 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194020+0000", 2024-09-15T03:41:22.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:41:22.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194026+0000", 2024-09-15T03:41:22.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:41:22.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194028+0000", 2024-09-15T03:41:22.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.475e-06 2024-09-15T03:41:22.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195860+0000", 2024-09-15T03:41:22.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018314609999999999 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195941+0000", 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1464999999999993e-05 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815378+0000", 2024-09-15T03:41:22.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61943704499999996 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815470+0000", 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1351e-05 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.816188+0000", 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071828500000000004 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.157 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:175 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.910648326, 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068787259000000003, 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.158 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 175 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194064+0000", 2024-09-15T03:41:22.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:41:22.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194065+0000", 2024-09-15T03:41:22.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5540000000000001e-06 2024-09-15T03:41:22.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194068+0000", 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0730000000000001e-06 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194071+0000", 2024-09-15T03:41:22.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4770000000000002e-06 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195541+0000", 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014705670000000001 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195649+0000", 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108057 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.262851+0000", 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067201307000000002 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.165 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 35/32 ECSubWriteReply(tid=30, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:22.166 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:22.166 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.908112003999999, 2024-09-15T03:41:22.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61874190500000004, 2024-09-15T03:41:22.166 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.166 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.167 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:22.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:22.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:22.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:22.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196603+0000", 2024-09-15T03:41:22.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7719999999999999e-06 2024-09-15T03:41:22.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814918+0000", 2024-09-15T03:41:22.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61831525899999995 2024-09-15T03:41:22.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814925+0000", 2024-09-15T03:41:22.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1629999999999999e-06 2024-09-15T03:41:22.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814972+0000", 2024-09-15T03:41:22.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7231999999999998e-05 2024-09-15T03:41:22.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815342+0000", 2024-09-15T03:41:22.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000369019 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:300 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:22.174 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1000748340000008, 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0215413659999999, 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 300 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004638+0000", 2024-09-15T03:41:22.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6020000000000001e-06 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004976+0000", 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033792499999999999 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004994+0000", 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7507e-05 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.005001+0000", 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008081+0000", 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003079675 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:22.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008085+0000", 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3420000000000001e-06 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026131+0000", 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0180457119999999 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026178+0000", 2024-09-15T03:41:22.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6953000000000003e-05 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:301 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0955605750000004, 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0170563379999999, 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 301 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:22.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009153+0000", 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4569999999999999e-06 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009921+0000", 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076796899999999997 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009955+0000", 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3696999999999999e-05 2024-09-15T03:41:22.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009971+0000", 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5441e-05 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010025+0000", 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4712000000000001e-05 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010029+0000", 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.63e-06 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026201+0000", 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016171661 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026207+0000", 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7710000000000001e-06 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:302 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0947250799999999, 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016248987, 2024-09-15T03:41:22.186 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 302 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7700000000000004e-07 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010036+0000", 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8857999999999999e-05 2024-09-15T03:41:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010045+0000", 2024-09-15T03:41:22.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6200000000000005e-06 2024-09-15T03:41:22.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010049+0000", 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3649999999999997e-06 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010226+0000", 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017731600000000001 2024-09-15T03:41:22.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010228+0000", 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3090000000000001e-06 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026231+0000", 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016003078 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026236+0000", 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6639999999999997e-06 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.192 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:303 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0946547239999997, 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016202901, 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 303 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:22.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1399999999999997e-07 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010061+0000", 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010068+0000", 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9779999999999999e-06 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010072+0000", 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0489999999999997e-06 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010289+0000", 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021628399999999999 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010290+0000", 2024-09-15T03:41:22.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.201e-06 2024-09-15T03:41:22.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026255+0000", 2024-09-15T03:41:22.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015965209 2024-09-15T03:41:22.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026260+0000", 2024-09-15T03:41:22.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6659999999999999e-06 2024-09-15T03:41:22.198 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:304 3.11s1 3.95fce6f1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:22.199 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0946358709999995, 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0162086930000001, 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 304 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.200 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:22.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:22.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010078+0000", 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.711e-06 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010082+0000", 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1320000000000004e-06 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010084+0000", 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5450000000000002e-06 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010311+0000", 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022657600000000001 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010314+0000", 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.565e-06 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026281+0000", 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015966967 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026284+0000", 2024-09-15T03:41:22.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 47/32 ECSubWriteReply(tid=66, last_complete=47'18, committed=1, applied=1))", 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0920786299999996, 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16187408, 2024-09-15T03:41:22.206 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:22.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3799999999999999e-07 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012636+0000", 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.345e-06 2024-09-15T03:41:22.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173941+0000", 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16130529900000001 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173963+0000", 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.243e-05 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174016+0000", 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2948000000000002e-05 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174507+0000", 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049081999999999999 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.210 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s0 3:cce4e8f8:::smithi10536496-4:head [create,setxattr Yt`Ot\\d[ZfNaQaqU[W\\PSa^pNWWUkki`hjlaoWcR (805) in=40b,setxattr ca[gmd`SV\\jp[aXclsea`kka[hPURW\\j[hao]pna (594) in=40b,setxattr jjQP\\pmOqcO\\iaNd[l\\^a]fXSRsgi\\sVYrQS[]ia (1614) in=40b,setxattr VeZgjt^aNniSnQhWPO[qm\\[itg[XR[[jXsbagns_ (454) in=40b,setxattr jath]ttaVem][bRrjP]pdOZWWo__SYO]VOaZhXRj (1858) in=40b,setxattr _oSdRYPilSVobVgWRbrP_piatjWaoYQ\\qcaVbh^g (138) in=40b,setxattr bTSZgV_k`ccTPieZXaVqe[OXkZefRqrsPPlisa^a (946) in=40b,setxattr OkXYkpcX_kP[TmZOosPhjlctOQRUagSbp`XaYrXW (910) in=40b,setxattr rjkSWecoRfpnaXdQmX`a`dPYt]P[_hRUfj`SdY[ (911) in=39b,setxattr OcknbT^]PQTVTj\\iodYl`oq[]Te]Us]e[nbQYY^N (1330) in=40b,setxattr r^qtXit^UbRgSb`o\\aoaSSnZqmZeWhhehTPZsgUO (418) in=40b,setxattr bnoZ`pV]YYNYqogeSaPofWR_PtahQZ_fR_h_nTnT (1190) in=40b,setxattr rjred]_aNiaaolkeaffPXTfZRd[[_PYZaUXNOUUt (33) in=40b,setxattr gnal\\XR[qfmfOecOmacYb_WWa\\W[rRPNWne`tlNT (1437) in=40b,setxattr sZmfXOTVQZ^VacNNaltSYOghkYNpdSfdZRhnXTn_ (4981) in=40b,setxattr TsTNdOVOpOPdoXYcpnamYadP^epnqllcbVsWNp^[ (17753) in=40b,setxattr PXkZNpT]nPaTrW]j`o_[RWiQrOrco]e_foQTXmO_ (1997) in=40b,setxattr sX[hspQmOVmae`epqt]pac_ZNQVn`YjUVU]QfSg^ (1051) in=40b,setxattr ^gV_afUghN\\gRs]pcaefnp[badg_a`]ROfd_ZZnW (5752) in=40b,setxattr d\\kRaqs`ZmPOe`_`ZimVtjOoQ[NieeSSZocTVNl_ (313) in=40b] snapc c=[c,b] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.0927703619999996, 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0346458860000001, 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011942+0000", 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2100000000000002e-07 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011959+0000", 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7248000000000002e-05 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011965+0000", 2024-09-15T03:41:22.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8880000000000002e-06 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011969+0000", 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8929999999999998e-06 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.012000+0000", 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1004999999999999e-05 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.012002+0000", 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.289e-06 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.035078+0000", 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0230762010000001 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.035294+0000", 2024-09-15T03:41:22.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000215625 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.036366+0000", 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00107211 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046017+0000", 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096507329999999999 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046134+0000", 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011691199999999999 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046587+0000", 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000453561 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.217 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:321 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.0215507920000002, 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11431013299999999, 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 321 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:22.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:41:22.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083167+0000", 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5169999999999999e-06 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083174+0000", 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9890000000000003e-06 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083178+0000", 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2220000000000004e-06 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.084760+0000", 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001582008 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.084856+0000", 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5461e-05 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085326+0000", 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046985000000000001 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.087826+0000", 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025000759999999999 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197378+0000", 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.109552474 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197471+0000", 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2888999999999998e-05 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:322 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.0215262589999998, 2024-09-15T03:41:22.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12070206999999999, 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 322 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:41:22.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083186+0000", 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.89e-07 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083188+0000", 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1009999999999999e-06 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083195+0000", 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083199+0000", 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.41e-06 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085640+0000", 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002441838 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085722+0000", 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1780999999999995e-05 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197721+0000", 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.111998507 2024-09-15T03:41:22.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197792+0000", 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0963000000000005e-05 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.203887+0000", 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060957049999999999 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 50/32 ECSubWriteReply(tid=75, last_complete=50'22, committed=1, applied=1))", 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.0177479739999997, 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.110727595, 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:22.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086965+0000", 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6109999999999999e-06 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197067+0000", 2024-09-15T03:41:22.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.110101613 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197072+0000", 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7940000000000002e-06 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197329+0000", 2024-09-15T03:41:22.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025732199999999999 2024-09-15T03:41:22.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197691+0000", 2024-09-15T03:41:22.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036200000000000002 2024-09-15T03:41:22.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.235 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 50/32 ECSubWriteReply(tid=80, last_complete=50'22, committed=1, applied=1))", 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.904106692, 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.82440747800000003, 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.91e-07 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200607+0000", 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:41:22.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024931+0000", 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.82432465200000005 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024938+0000", 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2129999999999996e-06 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024985+0000", 2024-09-15T03:41:22.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7713999999999998e-05 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.025012+0000", 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7058000000000001e-05 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.241 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:22.243 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:22.243 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:22.241+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:22.344 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:22.343+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:22.381 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:22.382 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:22.382 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:22.382 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:22.390 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:22.445 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:22.444+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:22.497 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:22.497 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:22.497 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:22.497 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:22.497 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:22.506 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:22.546 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:22.545+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=14, to_read={3:0e027997:::smithi10536496-12:head=0,4096,0}, subchunks={3:0e027997:::smithi10536496-12:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.565491256000001, 2024-09-15T03:41:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0095256449999999993, 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037696+0000", 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1408e-05 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046920+0000", 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092243440000000006 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046927+0000", 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8090000000000004e-06 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046974+0000", 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6869000000000001e-05 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047209+0000", 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235687 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:49 3.5s0 3:ac8f3d24:::smithi10536496-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.564722930999999, 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091978560000000008, 2024-09-15T03:41:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 49 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038454+0000", 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7099999999999999e-06 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038626+0000", 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017214699999999999 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038636+0000", 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0199e-05 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038641+0000", 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.685e-06 2024-09-15T03:41:22.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038679+0000", 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7731999999999997e-05 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038769+0000", 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0315999999999994e-05 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.039430+0000", 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00066085199999999997 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040743+0000", 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001313346 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047587+0000", 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068434860000000002 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047650+0000", 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3382999999999999e-05 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:50 3.5s0 3:ac8f3d24:::smithi10536496-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.564525883000002, 2024-09-15T03:41:22.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091805140000000007, 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 50 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038650+0000", 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038652+0000", 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.182e-06 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038656+0000", 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5090000000000004e-06 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038659+0000", 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4030000000000001e-06 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.039949+0000", 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001289884 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040025+0000", 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6005000000000002e-05 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:22.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00037365599999999999 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040954+0000", 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055585300000000003 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047765+0000", 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068113210000000004 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047830+0000", 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4237000000000002e-05 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:51 3.5s0 3:ac8f3d24:::smithi10536496-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:22.621 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.564500854999999, 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01164283, 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 51 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:22.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2599999999999998e-07 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038676+0000", 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.508e-06 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038680+0000", 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.653e-06 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038685+0000", 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9269999999999996e-06 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040567+0000", 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00188288 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040733+0000", 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016570300000000001 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048135+0000", 2024-09-15T03:41:22.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0074020889999999997 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048217+0000", 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2152999999999998e-05 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050317+0000", 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002099591 2024-09-15T03:41:22.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=16, reqid=client.4562.0:65, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.555638153, 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020348845000000001, 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5899999999999996e-07 2024-09-15T03:41:22.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047553+0000", 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6099999999999998e-05 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066726+0000", 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019172577 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066731+0000", 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3340000000000001e-06 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066780+0000", 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8708999999999997e-05 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066790+0000", 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9709999999999999e-06 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067856+0000", 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001066259 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067886+0000", 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9335999999999998e-05 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.631 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:69 3.5s0 3:af19ecb4:::smithi10536496-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.554592508999999, 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094136830000000008, 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 69 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048583+0000", 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:41:22.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048585+0000", 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4729999999999999e-06 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048592+0000", 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2360000000000001e-06 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048596+0000", 2024-09-15T03:41:22.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.092e-06 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051062+0000", 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024665440000000002 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051144+0000", 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1369e-05 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056556+0000", 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054122759999999997 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056628+0000", 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1699999999999995e-05 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057996+0000", 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001368534 2024-09-15T03:41:22.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=18, reqid=client.4562.0:86, at_version=35'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.535752738999999, 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057601141000000002, 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067423+0000", 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2499999999999995e-07 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067437+0000", 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4445e-05 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121493+0000", 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054055923999999998 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121501+0000", 2024-09-15T03:41:22.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8949999999999998e-06 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121585+0000", 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3502000000000001e-05 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121613+0000", 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8503000000000001e-05 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.124988+0000", 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033745590000000001 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125023+0000", 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5787999999999997e-05 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:114 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:22.641 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.505216918999999, 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093237049999999998, 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 114 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1899999999999998e-07 2024-09-15T03:41:22.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097960+0000", 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6130000000000001e-06 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097966+0000", 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3690000000000003e-06 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097970+0000", 2024-09-15T03:41:22.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0069999999999999e-06 2024-09-15T03:41:22.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099933+0000", 2024-09-15T03:41:22.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019621830000000002 2024-09-15T03:41:22.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.100014+0000", 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0852000000000004e-05 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104698+0000", 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046846140000000001 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104783+0000", 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4979000000000001e-05 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107282+0000", 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024986689999999998 2024-09-15T03:41:22.648 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=19, to_read={3:0215fa29:::smithi10536496-22:head=0,4096,0}, subchunks={3:0215fa29:::smithi10536496-22:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.481755184000001, 2024-09-15T03:41:22.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039885337, 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:22.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121442+0000", 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2169999999999999e-05 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160985+0000", 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039542246000000003 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160992+0000", 2024-09-15T03:41:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2840000000000001e-06 2024-09-15T03:41:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:22.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4248e-05 2024-09-15T03:41:22.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161305+0000", 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025900800000000002 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.9s4 35/32 ECSubWrite(tid=25, reqid=client.4562.0:161, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.439120359, 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075462681000000004, 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.655 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9299999999999998e-07 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164071+0000", 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5299e-05 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238920+0000", 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074849009999999994 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238927+0000", 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4039999999999998e-06 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238986+0000", 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9086000000000003e-05 2024-09-15T03:41:22.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238993+0000", 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3719999999999999e-06 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239496+0000", 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050214699999999999 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239517+0000", 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.177e-05 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:289 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.575117574, 2024-09-15T03:41:22.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016317213000000001, 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 289 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028059+0000", 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3969999999999999e-06 2024-09-15T03:41:22.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028066+0000", 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.627e-06 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028082+0000", 2024-09-15T03:41:22.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5406000000000001e-05 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028087+0000", 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4809999999999997e-06 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028181+0000", 2024-09-15T03:41:22.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3881999999999993e-05 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028280+0000", 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9135999999999994e-05 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028913+0000", 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00063293699999999997 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032481+0000", 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003567677 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044305+0000", 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011823828 2024-09-15T03:41:22.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044375+0000", 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9842e-05 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:290 3.5s0 3:a7103f89:::smithi10536496-39:head [rollback b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.575075180000001, 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016591980999999999, 2024-09-15T03:41:22.667 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 290 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028104+0000", 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.501e-06 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028112+0000", 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2379999999999997e-06 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028116+0000", 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3290000000000004e-06 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029591+0000", 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001475004 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029762+0000", 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000170557 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030258+0000", 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049562300000000005 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035019+0000", 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004761276 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044634+0000", 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096147730000000001 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044692+0000", 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8220000000000002e-05 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:291 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.575054012000001, 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019516250999999998, 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.674 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 291 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:22.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028123+0000", 2024-09-15T03:41:22.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1239999999999999e-06 2024-09-15T03:41:22.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028128+0000", 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5449999999999997e-06 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028132+0000", 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0090000000000001e-06 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030541+0000", 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024089659999999998 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030601+0000", 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0384000000000003e-05 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030921+0000", 2024-09-15T03:41:22.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031995500000000002 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035034+0000", 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0041124630000000002 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047571+0000", 2024-09-15T03:41:22.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012537510999999999 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047637+0000", 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5925000000000003e-05 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=22, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.566182873000001, 2024-09-15T03:41:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011072657, 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1899999999999999e-07 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036994+0000", 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047471+0000", 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010477119 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047475+0000", 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3989999999999997e-06 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047509+0000", 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3426000000000002e-05 2024-09-15T03:41:22.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.048065+0000", 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055612399999999996 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=23, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:22.687 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.550412017999999, 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92706636600000003, 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9599999999999999e-07 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052767+0000", 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8199999999999998e-06 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979592+0000", 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92682499399999996 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979604+0000", 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1387e-05 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979726+0000", 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122014 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979829+0000", 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103655 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.692 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:312 3.5s0 3:a1c61493:::smithi10536496-38:head [append 0~622592] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.057916+0000", 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.5452594020000001, 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0083523039999999996, 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 312 2024-09-15T03:41:22.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.057916+0000", 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.057916+0000", 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.057917+0000", 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.787e-06 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058334+0000", 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041702299999999998 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058348+0000", 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4079e-05 2024-09-15T03:41:22.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058355+0000", 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5200000000000003e-06 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058401+0000", 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5565999999999997e-05 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058545+0000", 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014402999999999999 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059495+0000", 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095007399999999997 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.066126+0000", 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066316860000000004 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.066198+0000", 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1996999999999995e-05 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.066268+0000", 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9542000000000006e-05 2024-09-15T03:41:22.698 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:313 3.5s0 3:a1c61493:::smithi10536496-38:head [append 0~147456] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:41:22.699 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.544217282, 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0098668620000000005, 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 313 2024-09-15T03:41:22.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058959+0000", 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.108e-06 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059032+0000", 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3231000000000004e-05 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059047+0000", 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4829e-05 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059053+0000", 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5910000000000001e-06 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.060287+0000", 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012338760000000001 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.060443+0000", 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015520099999999999 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.061055+0000", 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00061251899999999995 2024-09-15T03:41:22.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068673+0000", 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076178460000000002 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068754+0000", 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1104000000000001e-05 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068825+0000", 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0556999999999995e-05 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:314 3.5s0 3:a1c61493:::smithi10536496-38:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.5441141900000002, 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015366338, 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.706 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 314 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-07 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059066+0000", 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8790000000000002e-06 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059075+0000", 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6009999999999995e-06 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059079+0000", 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999997e-06 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.063568+0000", 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044885849999999998 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.063652+0000", 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4670000000000001e-05 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.064159+0000", 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050645400000000004 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068840+0000", 2024-09-15T03:41:22.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046815729999999996 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074353+0000", 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055131449999999997 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074427+0000", 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3767e-05 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.712 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:315 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.5440915159999999, 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018255761999999998, 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.713 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:22.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.714 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 315 2024-09-15T03:41:22.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.714 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059086+0000", 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1390000000000002e-06 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3779999999999998e-06 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059098+0000", 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.065037+0000", 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005938484 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.065126+0000", 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9343000000000002e-05 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074804+0000", 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096778360000000004 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074926+0000", 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122092 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.077339+0000", 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024133190000000001 2024-09-15T03:41:22.720 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:325 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.520165521, 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.078787472999999997, 2024-09-15T03:41:22.721 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 325 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083010+0000", 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.46e-07 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083012+0000", 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083020+0000", 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5769999999999996e-06 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083024+0000", 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.143e-06 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085552+0000", 2024-09-15T03:41:22.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025276999999999999 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085630+0000", 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7831999999999996e-05 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086020+0000", 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039002200000000001 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:22.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.090715+0000", 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046956599999999999 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.161716+0000", 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071000895999999994 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.161797+0000", 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0729000000000006e-05 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.726 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:22.727 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:22.727 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:22.728 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:22.729 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:22.728+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:22.828 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:22.828+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:22.844 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:22.844 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:22.845 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:22.845 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:22.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:22.930 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:22.929+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:22.956 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:22.956 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:22.956 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:22.956 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:22.957 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:22.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:23.031 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:23.030+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:23.066 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:41:23.067 INFO:tasks.rados.rados.0.smithi105.stdout:164: done (0 left) 2024-09-15T03:41:23.067 INFO:tasks.rados.rados.0.smithi105.stdout:165: snap_remove snap 10 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:7 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.053288981000001, 2024-09-15T03:41:23.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01304598, 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 7 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017045+0000", 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5040000000000001e-06 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017174+0000", 2024-09-15T03:41:23.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012955 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017203+0000", 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8303e-05 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017222+0000", 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9680000000000001e-05 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021260800000000001 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017608+0000", 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017282699999999999 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018409+0000", 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080141399999999997 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020887+0000", 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002477244 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030008+0000", 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0091209519999999999 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030088+0000", 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0897999999999997e-05 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.079 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:8 3.6s0 3:62396a93:::smithi10536496-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.053075327999998, 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013086021999999999, 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 8 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017258+0000", 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6300000000000001e-06 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017267+0000", 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4830000000000005e-06 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017303+0000", 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5326e-05 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017316+0000", 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3073e-05 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019094+0000", 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001778352 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0492e-05 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019834+0000", 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064979700000000003 2024-09-15T03:41:23.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021641+0000", 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018066779999999999 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030253+0000", 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086118240000000006 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030342+0000", 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9367e-05 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:9 3.6s0 3:62396a93:::smithi10536496-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.052941697000001, 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015701551000000001, 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 9 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017391+0000", 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.23e-07 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017394+0000", 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5999999999999998e-06 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017406+0000", 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1980999999999999e-05 2024-09-15T03:41:23.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8459e-05 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020095+0000", 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026602399999999999 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:23.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020181+0000", 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6468e-05 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030742+0000", 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010561054 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015357899999999999 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.033091+0000", 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021953469999999998 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.090 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.048800271000001, 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050576110000000001, 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:23.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021532+0000", 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021586+0000", 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4029000000000001e-05 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023870+0000", 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022843490000000002 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023879+0000", 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7509999999999997e-06 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024332+0000", 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00045252999999999998 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024353+0000", 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1382999999999998e-05 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026560+0000", 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002206679 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026589+0000", 2024-09-15T03:41:23.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9149000000000002e-05 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 34/32 ECSubWriteReply(tid=0, last_complete=35'1, committed=1, applied=1))", 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.023018+0000", 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.047313215999999, 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039769899999999997, 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023018+0000", 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023018+0000", 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023019+0000", 2024-09-15T03:41:23.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9099999999999999e-07 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023022+0000", 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8229999999999997e-06 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023524+0000", 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00050125099999999998 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026742+0000", 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032187169999999998 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026983+0000", 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024108699999999999 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026995+0000", 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1721e-05 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.098 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.024961+0000", 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.045370420000001, 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034933970000000001, 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024961+0000", 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024961+0000", 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024961+0000", 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9000000000000001e-07 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.025003+0000", 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1711000000000001e-05 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.025824+0000", 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000821461 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.025831+0000", 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6950000000000003e-06 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.025897+0000", 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6064999999999999e-05 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.025905+0000", 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2330000000000006e-06 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028428+0000", 2024-09-15T03:41:23.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025222790000000001 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028454+0000", 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6763000000000001e-05 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs2 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.027237+0000", 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.043094723999999, 2024-09-15T03:41:23.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003800941, 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.027237+0000", 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.027237+0000", 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.027237+0000", 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4999999999999999e-07 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.027254+0000", 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7643999999999999e-05 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029286+0000", 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020310770000000001 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029295+0000", 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7699999999999996e-06 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029360+0000", 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4306000000000004e-05 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029399+0000", 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9045000000000003e-05 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.031008+0000", 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001609276 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.031038+0000", 2024-09-15T03:41:23.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9672999999999999e-05 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:79 3.6s0 3:60903cd9:::smithi10536496-27:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.011152514999999, 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056797210000000004, 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.108 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 79 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:23.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059181+0000", 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.905e-06 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059725+0000", 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000544675 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059743+0000", 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059749+0000", 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8680000000000001e-06 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059823+0000", 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3856999999999994e-05 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059937+0000", 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113934 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.060672+0000", 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00073485 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062007+0000", 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001335433 2024-09-15T03:41:23.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064787+0000", 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027791140000000001 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064859+0000", 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2015999999999996e-05 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:80 3.6s0 3:60903cd9:::smithi10536496-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.010550200000001, 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052634420000000001, 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 80 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.114 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059782+0000", 2024-09-15T03:41:23.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059787+0000", 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9010000000000002e-06 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059797+0000", 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059802+0000", 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7310000000000001e-06 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061262+0000", 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014593609999999999 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061347+0000", 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5358999999999994e-05 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061756+0000", 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00040932300000000001 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062312+0000", 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055575999999999998 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064973+0000", 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026609950000000002 2024-09-15T03:41:23.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065045+0000", 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1703000000000001e-05 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:81 3.6s0 3:60903cd9:::smithi10536496-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.010505936000001, 2024-09-15T03:41:23.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055321019999999997, 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 81 2024-09-15T03:41:23.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059826+0000", 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4899999999999995e-07 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059830+0000", 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5769999999999998e-06 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059837+0000", 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4490000000000002e-06 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059841+0000", 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.765e-06 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062037+0000", 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002196506 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:23.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062113+0000", 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5740000000000003e-05 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065347+0000", 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003233648 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065358+0000", 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0868e-05 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs2 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.020828+0000", 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.049502969000001, 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035695380000000001, 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.020828+0000", 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.020828+0000", 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.020829+0000", 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9699999999999996e-07 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022016+0000", 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001187149 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022158+0000", 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014161499999999999 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022166+0000", 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9719999999999992e-06 2024-09-15T03:41:23.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022313+0000", 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014778600000000001 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022326+0000", 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2323e-05 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024363+0000", 2024-09-15T03:41:23.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020370409999999999 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024398+0000", 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5154999999999998e-05 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=60, reqid=client.4562.0:281, at_version=45'25, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:23.129 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.048567276, 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068718160000000002, 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:23.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4499999999999999e-07 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021848+0000", 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4214000000000004e-05 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024178+0000", 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023298640000000001 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024186+0000", 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3379999999999999e-06 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024242+0000", 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5829999999999999e-05 2024-09-15T03:41:23.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024252+0000", 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028612+0000", 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043595680000000003 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028636+0000", 2024-09-15T03:41:23.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3896999999999998e-05 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.045087197999999, 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087088040000000005, 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025245+0000", 2024-09-15T03:41:23.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0600000000000001e-07 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025397+0000", 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015260800000000001 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026753+0000", 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013562590000000001 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026763+0000", 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7340000000000003e-06 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026806+0000", 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3254000000000003e-05 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026816+0000", 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0115000000000001e-05 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033305+0000", 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00648871 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033953+0000", 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064781799999999998 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.138 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.042609703, 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006712165, 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.139 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:23.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027799+0000", 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6990999999999995e-05 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028647+0000", 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00084766699999999997 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028655+0000", 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6769999999999995e-06 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028695+0000", 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0281000000000003e-05 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028705+0000", 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7140000000000005e-06 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034407+0000", 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.005702141 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034434+0000", 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7005e-05 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.042528801, 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066515380000000002, 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027818+0000", 2024-09-15T03:41:23.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4961e-05 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029147+0000", 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001328842 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029156+0000", 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8710000000000003e-06 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029196+0000", 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9928999999999997e-05 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029204+0000", 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5809999999999997e-06 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034438+0000", 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052331399999999998 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034454+0000", 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6558e-05 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.5s5 45/44 ECSubWrite(tid=20, reqid=client.4562.0:289, at_version=45'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.029807+0000", 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.040524379000001, 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046313489999999999, 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.151 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029807+0000", 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029807+0000", 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029808+0000", 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7400000000000003e-07 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029831+0000", 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2846000000000001e-05 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029865+0000", 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4399e-05 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029871+0000", 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.348e-06 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029921+0000", 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0067e-05 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029931+0000", 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034416+0000", 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044853870000000004 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034438+0000", 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1957999999999999e-05 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.5s5 45/44 ECSubWrite(tid=21, reqid=client.4562.0:290, at_version=45'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.031032+0000", 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.039299623, 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044133840000000002, 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.156 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031032+0000", 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031032+0000", 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031032+0000", 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.61e-07 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031046+0000", 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4024e-05 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031329+0000", 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00028257299999999999 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031336+0000", 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9060000000000002e-06 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031392+0000", 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5306999999999997e-05 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031401+0000", 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2180000000000005e-06 2024-09-15T03:41:23.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.035426+0000", 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040246539999999999 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.035445+0000", 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9541000000000001e-05 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.5s5 45/44 ECSubWrite(tid=22, reqid=client.4562.0:291, at_version=45'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.031052+0000", 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.039279763, 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044099009999999999, 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031052+0000", 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031052+0000", 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031052+0000", 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2599999999999999e-07 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031069+0000", 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6914000000000001e-05 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031660+0000", 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00059175199999999997 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031670+0000", 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3309999999999993e-06 2024-09-15T03:41:23.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033169+0000", 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014996230000000001 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033178+0000", 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9889999999999998e-06 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.035450+0000", 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002271435 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.035462+0000", 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1731e-05 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.165 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.8s1 45/44 ECSubWrite(tid=22, reqid=client.4562.0:286, at_version=45'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.031476+0000", 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.038855302, 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044107720000000003, 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.166 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031476+0000", 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031476+0000", 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031477+0000", 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5000000000000002e-07 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.032027+0000", 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055040000000000004 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.032065+0000", 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8112999999999999e-05 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.032073+0000", 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1359999999999999e-06 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.032123+0000", 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0986999999999998e-05 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.032143+0000", 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9281999999999999e-05 2024-09-15T03:41:23.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.035865+0000", 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037224659999999998 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.035887+0000", 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1637999999999998e-05 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(osd.14.22:2 3.6s0 3:62396a93:::smithi10536496-3:head [assert-version v8,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.026825917, 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010013731999999999, 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043507+0000", 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9290000000000001e-06 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043518+0000", 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0934000000000001e-05 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043540+0000", 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1418000000000001e-05 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043548+0000", 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.602e-06 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043611+0000", 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3034000000000007e-05 2024-09-15T03:41:23.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043784+0000", 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017247700000000001 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.053519+0000", 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097353379999999996 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.174 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:23.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:23.176 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:23.175+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:23.177 INFO:teuthology.orchestra.run.smithi105.stderr:Nothing to do for non-filestore backend 2024-09-15T03:41:23.277 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:23.277+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:23.320 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.320 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:23.320 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:23.320 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:23.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:23.378 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:23.377+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:23.433 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.433 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:23.434 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:23.434 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:23.434 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:23.439 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:23.478 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:23.478+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=4, last_complete=34'2, committed=1, applied=1))", 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.507801213, 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049531878000000001, 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036255+0000", 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6899999999999999e-07 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.036257+0000", 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0109999999999999e-06 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085680+0000", 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049422928999999997 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085688+0000", 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9510000000000006e-06 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085748+0000", 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0078999999999999e-05 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085787+0000", 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8739000000000003e-05 2024-09-15T03:41:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:101 3.bs0 3:db73af7f:::smithi10536496-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.461531944000001, 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060658139999999999, 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 101 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082525+0000", 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082528+0000", 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0989999999999999e-06 2024-09-15T03:41:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082540+0000", 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2673999999999999e-05 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082544+0000", 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9559999999999999e-06 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083692+0000", 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011471949999999999 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083836+0000", 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014488899999999999 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084281+0000", 2024-09-15T03:41:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044464500000000002 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085132+0000", 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00085125900000000004 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143106+0000", 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057973867999999998 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143182+0000", 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5862e-05 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:102 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.461490334000001, 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100342765, 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 102 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082569+0000", 2024-09-15T03:41:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2279999999999998e-06 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082577+0000", 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9319999999999996e-06 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082584+0000", 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8009999999999998e-06 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084479+0000", 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018943709999999999 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084576+0000", 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7615000000000006e-05 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143227+0000", 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058651048999999997 2024-09-15T03:41:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143302+0000", 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4547000000000003e-05 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182908+0000", 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039606548999999998 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=25, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.458398160000002, 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057539988, 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.98e-07 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085660+0000", 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.902e-06 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142986+0000", 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057326329000000002 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142992+0000", 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-09-15T03:41:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143038+0000", 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6576999999999998e-05 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143198+0000", 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015933200000000001 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:111 3.7s0 3:edf86ef6:::smithi10536496-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.449674886, 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050567135999999999, 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 111 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094382+0000", 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094384+0000", 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7489999999999999e-06 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094393+0000", 2024-09-15T03:41:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2500000000000006e-06 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094397+0000", 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8040000000000001e-06 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002394565 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096872+0000", 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0492000000000001e-05 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105071+0000", 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081983339999999998 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105164+0000", 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3202999999999995e-05 2024-09-15T03:41:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.144948+0000", 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039784183000000001 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:140 3.bs0 3:dc608269:::smithi10536496-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.139972+0000", 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.404083530000001, 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041721282999999998, 2024-09-15T03:41:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 140 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139972+0000", 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139972+0000", 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139973+0000", 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139975+0000", 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0169999999999999e-06 2024-09-15T03:41:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139979+0000", 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8210000000000003e-06 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.139981+0000", 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2620000000000001e-06 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141767+0000", 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001785479 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.141858+0000", 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1113999999999995e-05 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142297+0000", 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043900500000000002 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143201+0000", 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00090441899999999995 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.181615+0000", 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038414306000000002 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.181694+0000", 2024-09-15T03:41:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8233999999999997e-05 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:141 3.bs0 3:dc608269:::smithi10536496-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.404051853999999, 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055455817999999997, 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 141 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6600000000000003e-07 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140006+0000", 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5409999999999999e-06 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140011+0000", 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.843e-06 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140013+0000", 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.621e-06 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002515181 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142605+0000", 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6328999999999994e-05 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182018+0000", 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039413493000000001 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182116+0000", 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8159000000000007e-05 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.195460+0000", 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013343384999999999 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpReadReply(3.bs0 35/32 ECSubReadReply(tid=35, attrs_read=0))", 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.143734+0000", 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.400321865999999, 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039182792000000001, 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143734+0000", 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143734+0000", 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143735+0000", 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1699999999999997e-07 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143746+0000", 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0986e-05 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182781+0000", 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039034916000000003 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182786+0000", 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7729999999999996e-06 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182814+0000", 2024-09-15T03:41:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8034999999999998e-05 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182917+0000", 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102365 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:273 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~802816] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.517137809999999, 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0316972600000001, 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026920+0000", 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5629999999999999e-06 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027741+0000", 2024-09-15T03:41:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082160000000000002 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027764+0000", 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2495000000000001e-05 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027772+0000", 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027837+0000", 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5062999999999997e-05 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027841+0000", 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043211+0000", 2024-09-15T03:41:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.015370181 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043389+0000", 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017807500000000001 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044733+0000", 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013438510000000001 2024-09-15T03:41:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048847+0000", 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041134250000000004 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058556+0000", 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097088929999999997 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058615+0000", 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9623999999999997e-05 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:274 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~376832] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.514657735, 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.029738034, 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 274 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029399+0000", 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6099999999999999e-07 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029738+0000", 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00033941400000000001 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029756+0000", 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524e-05 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029763+0000", 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9609999999999999e-06 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029831+0000", 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8169000000000004e-05 2024-09-15T03:41:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029833+0000", 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045818+0000", 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0159847550000001 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045920+0000", 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102085 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.046687+0000", 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00076710099999999996 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049023+0000", 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023356919999999999 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059069+0000", 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010046019999999999 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059136+0000", 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7156999999999994e-05 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:275 3.17s0 3:ec08d906:::smithi10536496-14:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.514265850999999, 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0294973439999999, 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 275 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029791+0000", 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029807+0000", 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6107e-05 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029828+0000", 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1475999999999999e-05 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029838+0000", 2024-09-15T03:41:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9879999999999999e-06 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029885+0000", 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6887e-05 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029887+0000", 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5030000000000001e-06 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047307+0000", 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0174202429999999 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047434+0000", 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012744699999999999 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048033+0000", 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00059865199999999997 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049643+0000", 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016096159999999999 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059241+0000", 2024-09-15T03:41:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095980579999999996 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059287+0000", 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6726e-05 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:276 3.17s0 3:ec08d906:::smithi10536496-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.514199290000001, 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0302367509999999, 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 276 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029862+0000", 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4259999999999996e-06 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029875+0000", 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3132999999999999e-05 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029880+0000", 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029913+0000", 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3079e-05 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029914+0000", 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.164e-06 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048304+0000", 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.018390516 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048405+0000", 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010111300000000001 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059337+0000", 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010931523 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059408+0000", 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0962000000000003e-05 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.060093+0000", 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068532499999999997 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:279 3.17s0 3:eb414ad4:::smithi10536496-35:head [copy-from ver 4] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.011839+0000", 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.532216772, 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050037958, 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 279 2024-09-15T03:41:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011839+0000", 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011839+0000", 2024-09-15T03:41:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011842+0000", 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4499999999999998e-06 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011855+0000", 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3529e-05 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011876+0000", 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0809e-05 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011887+0000", 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0648e-05 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012113+0000", 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00022619599999999999 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012124+0000", 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0978000000000001e-05 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016030+0000", 2024-09-15T03:41:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039067399999999997 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016035+0000", 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6539999999999998e-06 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048493+0000", 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032458289000000001 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048644+0000", 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015087999999999999 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.057407+0000", 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087630199999999998 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058748+0000", 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013411849999999999 2024-09-15T03:41:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.061839+0000", 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030909940000000001 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.061877+0000", 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7586000000000002e-05 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:280 3.17s0 3:eb414ad4:::smithi10536496-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.011912+0000", 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.532143877999999, 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046547195999999999, 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 280 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011912+0000", 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011912+0000", 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011913+0000", 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1500000000000003e-07 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011916+0000", 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.67e-06 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011927+0000", 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0108e-05 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.011932+0000", 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2979999999999999e-06 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012189+0000", 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00025723199999999998 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.012196+0000", 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5760000000000002e-06 2024-09-15T03:41:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016057+0000", 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038613499999999999 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.016058+0000", 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2500000000000005e-07 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048889+0000", 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032831030999999997 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048942+0000", 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2948999999999997e-05 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058223+0000", 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0092811779999999993 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058328+0000", 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010461199999999999 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058459+0000", 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013165200000000001 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.5433773679999998, 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.030189832, 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000680+0000", 2024-09-15T03:41:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000683+0000", 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6610000000000002e-06 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000708+0000", 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5165999999999999e-05 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000714+0000", 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6250000000000004e-06 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005624+0000", 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049095390000000001 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005627+0000", 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-06 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030812+0000", 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025185383 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030868+0000", 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6032000000000002e-05 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.5433217989999992, 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301912289999999, 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000735+0000", 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000739+0000", 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0119999999999999e-06 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000747+0000", 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1159999999999999e-06 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000751+0000", 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048937850000000003 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5700000000000002e-07 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030914+0000", 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252686520000001 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030925+0000", 2024-09-15T03:41:23.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1248e-05 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.5432133980000007, 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301360509999999, 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000843+0000", 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000856+0000", 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2296999999999999e-05 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000869+0000", 2024-09-15T03:41:23.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3045e-05 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000878+0000", 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6749999999999997e-06 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005662+0000", 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047836379999999998 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005663+0000", 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3300000000000002e-07 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030967+0000", 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025303984 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030978+0000", 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1972e-05 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:298 3.7s0 3:e4711aff:::smithi10536496-25:head [rollback c] snapc c=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000944+0000", 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.5431117729999997, 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027237540000000001, 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 298 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000944+0000", 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000944+0000", 2024-09-15T03:41:23.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000945+0000", 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8800000000000002e-07 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000950+0000", 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7379999999999999e-06 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000959+0000", 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3999999999999992e-06 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000963+0000", 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4710000000000001e-06 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001090+0000", 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012669299999999999 2024-09-15T03:41:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001096+0000", 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2580000000000001e-06 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012316+0000", 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011219303 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012440+0000", 2024-09-15T03:41:23.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012406099999999999 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.013822+0000", 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001382168 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015672+0000", 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018506040000000001 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028143+0000", 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01247068 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028182+0000", 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8575999999999998e-05 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:299 3.7s0 3:e4711aff:::smithi10536496-25:head [append 0~0] snapc c=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000968+0000", 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.5430876550000008, 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027322619999999999, 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 299 2024-09-15T03:41:23.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000968+0000", 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000968+0000", 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000969+0000", 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.32e-07 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000971+0000", 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2010000000000002e-06 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000977+0000", 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.601e-06 2024-09-15T03:41:23.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001007+0000", 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9858000000000001e-05 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001120+0000", 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112538 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:23.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.001127+0000", 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1319999999999996e-06 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.014248+0000", 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013121224000000001 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.014345+0000", 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7167e-05 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.014832+0000", 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00048681899999999999 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016737+0000", 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019046130000000001 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028252+0000", 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01151509 2024-09-15T03:41:23.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028291+0000", 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8945000000000001e-05 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.4s2 REQUEST e53)", 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.767783712, 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27094768699999999, 2024-09-15T03:41:23.663 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776274+0000", 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.007e-06 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776283+0000", 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9870000000000005e-06 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047099+0000", 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.270815584 2024-09-15T03:41:23.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047106+0000", 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7249999999999997e-06 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047165+0000", 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8955000000000001e-05 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047186+0000", 2024-09-15T03:41:23.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047220+0000", 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4156999999999998e-05 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.667 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:23.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:23.669 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:23.668+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:23.695 DEBUG:teuthology.orchestra.run.smithi105:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool rbd 2024-09-15T03:41:23.769 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:23.768+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:23.772 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.772 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:23.772 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:23.772 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:23.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:23.870 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:23.869+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:23.883 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.883 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:23.883 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:23.883 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:23.883 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:23.891 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:23.970 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:23.970+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:23.994 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.9s5 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.973152728999999, 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0065251390000000001, 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:23.995 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:23.996 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019746+0000", 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8999999999999996e-07 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019922+0000", 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017537400000000001 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019959+0000", 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7289999999999997e-05 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019969+0000", 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020062+0000", 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2602000000000005e-05 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020072+0000", 2024-09-15T03:41:23.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048e-05 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026239+0000", 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061667700000000002 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026271+0000", 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2231999999999999e-05 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:23.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.9s5 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.023431+0000", 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.969467290000001, 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028587669999999999, 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023431+0000", 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023431+0000", 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023432+0000", 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.082e-06 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023453+0000", 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0424999999999998e-05 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023500+0000", 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7051000000000001e-05 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023511+0000", 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0842000000000001e-05 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.002 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023574+0000", 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3046000000000006e-05 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023584+0000", 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9890000000000004e-06 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026276+0000", 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026923680000000001 2024-09-15T03:41:24.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026290+0000", 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3964e-05 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.029621+0000", 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.963277830999999, 2024-09-15T03:41:24.004 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025459609999999998, 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029621+0000", 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029621+0000", 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029622+0000", 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1349999999999999e-06 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029829+0000", 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000207407 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029922+0000", 2024-09-15T03:41:24.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3127000000000005e-05 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.029934+0000", 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1165000000000001e-05 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030000+0000", 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6540999999999999e-05 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030010+0000", 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8870000000000004e-06 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.032135+0000", 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021250190000000001 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.032167+0000", 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1680000000000002e-05 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.008 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs1 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.971734747999999, 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003379225, 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.009 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021164+0000", 2024-09-15T03:41:24.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0400000000000002e-07 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022429+0000", 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001264734 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022492+0000", 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2748000000000003e-05 2024-09-15T03:41:24.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022499+0000", 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0010000000000004e-06 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022553+0000", 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4358999999999999e-05 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.012 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022564+0000", 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0618000000000001e-05 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024511+0000", 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019475479999999999 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024543+0000", 2024-09-15T03:41:24.013 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1912999999999998e-05 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.026458+0000", 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.966440597, 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026774849999999998, 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.014 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026458+0000", 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026458+0000", 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026459+0000", 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9100000000000003e-07 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026626+0000", 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000167617 2024-09-15T03:41:24.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026745+0000", 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011841300000000001 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026753+0000", 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000001e-06 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026888+0000", 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134313 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026899+0000", 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1124000000000001e-05 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029109+0000", 2024-09-15T03:41:24.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022096479999999998 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029136+0000", 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6919000000000001e-05 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.026631+0000", 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.966267387, 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037504040000000002, 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.020 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026631+0000", 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026631+0000", 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026632+0000", 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0900000000000002e-07 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026675+0000", 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3244999999999997e-05 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.022 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028134+0000", 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001459382 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028144+0000", 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6479999999999998e-06 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028494+0000", 2024-09-15T03:41:24.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00035034600000000002 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028505+0000", 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030354+0000", 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018489999999999999 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030382+0000", 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7605000000000001e-05 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.029234+0000", 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.963664616999999, 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024709860000000001, 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.025 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029234+0000", 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029234+0000", 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029235+0000", 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0500000000000003e-07 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029253+0000", 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7982000000000002e-05 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030584+0000", 2024-09-15T03:41:24.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001331884 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030591+0000", 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9280000000000004e-06 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030648+0000", 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7009999999999998e-05 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.030659+0000", 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0512e-05 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031682+0000", 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001023396 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.031705+0000", 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2668999999999999e-05 2024-09-15T03:41:24.029 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.5s1 47/44 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=47'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.002555+0000", 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.9903432260000002, 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006765091, 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.030 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002555+0000", 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002555+0000", 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002556+0000", 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2000000000000001e-07 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002577+0000", 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1115999999999999e-05 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002651+0000", 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3701999999999993e-05 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.002657+0000", 2024-09-15T03:41:24.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5390000000000004e-06 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.003242+0000", 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058518600000000002 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.003252+0000", 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9930000000000007e-06 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.009295+0000", 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060427470000000002 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.009320+0000", 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5488000000000001e-05 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.034 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=74, reqid=client.4562.0:297, at_version=47'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.9806549249999996, 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004343876, 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012244+0000", 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3e-07 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012269+0000", 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5018999999999999e-05 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015244+0000", 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029748069999999999 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015252+0000", 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5709999999999998e-06 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015331+0000", 2024-09-15T03:41:24.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8817000000000001e-05 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015342+0000", 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0570999999999999e-05 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016564+0000", 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012226870000000001 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016588+0000", 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3274000000000001e-05 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=76, reqid=client.4562.0:299, at_version=47'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.015200+0000", 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.9776981790000008, 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024177439999999999, 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.039 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015200+0000", 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015200+0000", 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015201+0000", 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2300000000000002e-07 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015225+0000", 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.393e-05 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015973+0000", 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074839000000000004 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.015980+0000", 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016374+0000", 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00039373899999999999 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.016384+0000", 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0451e-05 2024-09-15T03:41:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.017597+0000", 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001212393 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.017618+0000", 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1313000000000001e-05 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=77, reqid=client.4562.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=47'19))", 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.9644612939999995, 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92662449099999999, 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028438+0000", 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2099999999999996e-07 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028447+0000", 2024-09-15T03:41:24.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8759999999999994e-06 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954390+0000", 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92594294600000004 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954412+0000", 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2218e-05 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954469+0000", 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7000999999999999e-05 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954479+0000", 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5489999999999995e-06 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955037+0000", 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055842499999999998 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955062+0000", 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4754999999999999e-05 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:310 3.as0 3.bc9e946a (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e48)", 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.9823101639999994, 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99312189699999998, 2024-09-15T03:41:24.048 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 310 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010590+0000", 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.581e-06 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010594+0000", 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.98e-06 2024-09-15T03:41:24.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010608+0000", 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4251e-05 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010617+0000", 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7819999999999999e-06 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010695+0000", 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8144000000000003e-05 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010699+0000", 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8659999999999999e-06 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003696+0000", 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99299729199999998 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003710+0000", 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4001e-05 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.988217047, 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048967167, 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.053 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004683+0000", 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2899999999999999e-06 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004689+0000", 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9229999999999999e-06 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004703+0000", 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004719+0000", 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6095000000000001e-05 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005604+0000", 2024-09-15T03:41:24.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00088508100000000002 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005608+0000", 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.247e-06 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030447+0000", 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.024839515 2024-09-15T03:41:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030545+0000", 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7453000000000004e-05 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.053649+0000", 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023104197999999999 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.988166455, 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.049664036, 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004733+0000", 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0100000000000004e-07 2024-09-15T03:41:24.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004737+0000", 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9049999999999999e-06 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004745+0000", 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7540000000000006e-06 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004752+0000", 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9000000000000006e-06 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005631+0000", 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087836100000000003 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005632+0000", 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.47e-07 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030868+0000", 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252366669999999 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030963+0000", 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4665000000000003e-05 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054396+0000", 2024-09-15T03:41:24.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023433236 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.9881405000000001, 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0501537240000001, 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:24.064 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004759+0000", 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.82e-07 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004762+0000", 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2629999999999999e-06 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004769+0000", 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0260000000000002e-06 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004780+0000", 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0569e-05 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005657+0000", 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087720500000000004 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005658+0000", 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.159e-06 2024-09-15T03:41:24.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031236+0000", 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0255778339999999 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031320+0000", 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4217999999999999e-05 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054912+0000", 2024-09-15T03:41:24.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023591867999999998 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpRead(3.19s0 50/49 ECSubRead(tid=0, to_read={3:9a083222:::smithi10536496-8:head=0,245760,0}, subchunks={3:9a083222:::smithi10536496-8:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.9621713849999995, 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022999359999999998, 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030727+0000", 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030799+0000", 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2212999999999995e-05 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030809+0000", 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6290000000000005e-06 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.032658+0000", 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018488949999999999 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.033027+0000", 2024-09-15T03:41:24.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00036919900000000001 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs1 50/32 ECSubWrite(tid=74, reqid=client.4562.0:320, at_version=50'21, trim_to=0'0, roll_forward_to=47'18))", 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.084148+0000", 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.9087506279999999, 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025941219999999999, 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084148+0000", 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084148+0000", 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084149+0000", 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9800000000000003e-07 2024-09-15T03:41:24.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085045+0000", 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000896311 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085097+0000", 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2163999999999997e-05 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085104+0000", 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.878e-06 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085160+0000", 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5855e-05 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085170+0000", 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0254e-05 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086721+0000", 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001551092 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086742+0000", 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0869999999999998e-05 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es4 50/32 ECSubWrite(tid=43, reqid=client.4562.0:354, at_version=50'19, trim_to=0'0, roll_forward_to=50'16))", 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.162811+0000", 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.8300874790000004, 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.005319921, 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.162811+0000", 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.162811+0000", 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.162812+0000", 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.37e-07 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.163028+0000", 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021615600000000001 2024-09-15T03:41:24.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167226+0000", 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004198022 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167232+0000", 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9399999999999999e-06 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167288+0000", 2024-09-15T03:41:24.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6029000000000002e-05 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.167298+0000", 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0186999999999999e-05 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.168108+0000", 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00081004799999999995 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.168131+0000", 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2802e-05 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs1 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.024245+0000", 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.9686535699999999, 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054854049999999996, 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024245+0000", 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024245+0000", 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024245+0000", 2024-09-15T03:41:24.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3300000000000001e-07 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024264+0000", 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8995e-05 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024997+0000", 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00073246599999999998 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.025004+0000", 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2960000000000001e-06 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.025084+0000", 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0222999999999994e-05 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.025127+0000", 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3139999999999997e-05 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.029710+0000", 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004582371 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.029730+0000", 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0681000000000001e-05 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.ds5 56/51 ECSubWrite(tid=34, reqid=osd.7.0:34, at_version=56'16, trim_to=0'0, roll_forward_to=52'14))", 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:23.066287+0000", 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.92661146400000005, 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037982440000000001, 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066287+0000", 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066287+0000", 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066287+0000", 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4499999999999998e-07 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066310+0000", 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2320999999999999e-05 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066812+0000", 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00050190599999999999 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066850+0000", 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8420000000000001e-05 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.067359+0000", 2024-09-15T03:41:24.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00050941300000000001 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.067378+0000", 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8193999999999999e-05 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.070064+0000", 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026863719999999998 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:24.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.070085+0000", 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1373e-05 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:24.092 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:24.092 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:24.093 INFO:tasks.rados.rados.0.smithi105.stdout:165: done (0 left) 2024-09-15T03:41:24.094 INFO:tasks.rados.rados.0.smithi105.stdout:166: snap_remove snap 5 2024-09-15T03:41:24.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:24.095 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:41:24.096 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:24.093+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:24.195 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:24.195+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:24.238 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:24.238 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:24.238 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:24.238 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:24.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:24.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:24.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:24.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:24.286 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:24.296 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:24.296+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:24.307 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:24.307 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:24.307 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:24.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:24.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:24.348 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:24.349 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:24.367 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:24.367 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:24.368 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:24.368 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:24.368 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:24.369 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:24.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:24.376 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:24.389 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:24.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:24.398 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:24.397+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:24.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:24.409 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:24.429 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:24.430 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:24.450 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:24.450 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:24.470 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:24.471 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:24.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:24.490 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:24.491 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:24.492 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:24.499 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:24.498+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:24.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:24.513 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:24.513 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:24.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:24.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:24.554 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:24.555 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:24.558 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:24.600 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:24.599+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:24.617 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:24.617 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:24.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:24.701 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:24.700+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:24.732 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:24.732 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:24.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:24.802 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:24.801+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:24.834 INFO:teuthology.orchestra.run.smithi105.stderr:Nothing to do for non-filestore backend 2024-09-15T03:41:24.844 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:24.844 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:24.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:24.903 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:24.902+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:24.955 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:24.955 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:24.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:25.007 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:25.005+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:25.066 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.066 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.071 INFO:tasks.rados.rados.0.smithi105.stdout:166: done (0 left) 2024-09-15T03:41:25.071 INFO:tasks.rados.rados.0.smithi105.stdout:167: snap_create 2024-09-15T03:41:25.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:25.105 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:25.105+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:25.206 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:25.206+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:25.218 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.218 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:25.307 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:25.306+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:25.328 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.328 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:25.356 DEBUG:teuthology.orchestra.run.smithi105:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool unique_pool_0 2024-09-15T03:41:25.408 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:25.407+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:25.437 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.438 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:25.509 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:25.508+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:25.547 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.548 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:25.610 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:25.609+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:25.657 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.658 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:25.711 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:25.709+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:25.767 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.768 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:25.812 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:25.812+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:25.913 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:25.913+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:25.917 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:25.918 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:25.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:26.013 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:26.013+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:26.042 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.042 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:26.076 INFO:tasks.rados.rados.0.smithi105.stdout:167: done (0 left) 2024-09-15T03:41:26.076 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 4 current snap is 12 2024-09-15T03:41:26.076 INFO:tasks.rados.rados.0.smithi105.stdout:168: seq_num 85 ranges {65536=671744,737280=770048} 2024-09-15T03:41:26.080 INFO:tasks.rados.rados.0.smithi105.stdout:168: writing smithi10536496-4 from 65536 to 737280 tid 1 2024-09-15T03:41:26.084 INFO:tasks.rados.rados.0.smithi105.stdout:168: writing smithi10536496-4 from 737280 to 1507328 tid 2 2024-09-15T03:41:26.085 INFO:tasks.rados.rados.0.smithi105.stdout:169: snap_create 2024-09-15T03:41:26.088 INFO:tasks.rados.rados.0.smithi105.stdout:168: finishing write tid 1 to smithi10536496-4 2024-09-15T03:41:26.091 INFO:tasks.rados.rados.0.smithi105.stdout:168: finishing write tid 2 to smithi10536496-4 2024-09-15T03:41:26.095 INFO:tasks.rados.rados.0.smithi105.stdout:168: finishing write tid 3 to smithi10536496-4 2024-09-15T03:41:26.096 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:41:26.099 INFO:tasks.rados.rados.0.smithi105.stdout:168: finishing write tid 4 to smithi10536496-4 2024-09-15T03:41:26.099 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 29 (ObjNum 85 snap 12 seq_num 85) dirty exists 2024-09-15T03:41:26.099 INFO:tasks.rados.rados.0.smithi105.stdout:168: left oid 4 (ObjNum 85 snap 12 seq_num 85) 2024-09-15T03:41:26.115 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:26.114+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:26.192 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.192 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:26.244 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:26.243+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:26.298 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.298 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:26.317 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:26.316+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:26.418 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:26.416+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:26.444 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.444 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.453 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:26.492 INFO:teuthology.orchestra.run.smithi105.stderr:Nothing to do for non-filestore backend 2024-09-15T03:41:26.519 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:26.518+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:26.552 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.552 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:26.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:26.644 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:26.643+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:26.690 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:26.691 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:26.698 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.698 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:26.711 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:26.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:26.720 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:26.719+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:26.732 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:26.732 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:26.732 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:26.752 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:26.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:26.772 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:26.772 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:26.792 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:26.792 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:26.812 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:26.812 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:26.821 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:26.820+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:26.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:26.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:26.845 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:26.845 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:26.852 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:26.853 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:26.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:26.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:26.873 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:26.893 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:26.893 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:26.913 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:26.913 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:26.921 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:26.919+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:26.932 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:26.933 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:26.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:26.932+0000 7f88caffd640 0 -- 172.21.15.5:0/3075290431 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f88cc192cb0 msgr2=0x7f88cc195090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T03:41:26.991 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:26.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:26.991 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:26.991 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:26.991 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:26.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:27.008 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.3 2024-09-15T03:41:27.008 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 show-label 2024-09-15T03:41:27.043 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:27.042+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:27.077 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:41:27.078 INFO:tasks.rados.rados.0.smithi105.stdout:168: done (1 left) 2024-09-15T03:41:27.078 INFO:tasks.rados.rados.0.smithi105.stdout:169: done (0 left) 2024-09-15T03:41:27.078 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 15 current snap is 13 2024-09-15T03:41:27.078 INFO:tasks.rados.rados.0.smithi105.stdout:170: seq_num 86 ranges {245760=688128,933888=540672,1474560=229376} 2024-09-15T03:41:27.083 INFO:tasks.rados.rados.0.smithi105.stdout:170: writing smithi10536496-15 from 245760 to 933888 tid 1 2024-09-15T03:41:27.086 INFO:tasks.rados.rados.0.smithi105.stdout:170: writing smithi10536496-15 from 933888 to 1474560 tid 2 2024-09-15T03:41:27.088 INFO:tasks.rados.rados.0.smithi105.stdout:170: writing smithi10536496-15 from 1474560 to 1703936 tid 3 2024-09-15T03:41:27.088 INFO:tasks.rados.rados.0.smithi105.stdout:171: read oid 48 snap -1 2024-09-15T03:41:27.088 INFO:tasks.rados.rados.0.smithi105.stdout:171: expect deleted 2024-09-15T03:41:27.088 INFO:tasks.rados.rados.0.smithi105.stdout:172: snap_create 2024-09-15T03:41:27.088 INFO:teuthology.orchestra.run.smithi105.stdout:inferring bluefs devices from bluestore path 2024-09-15T03:41:27.088 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "/var/lib/ceph/osd/ceph-3/block": { 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_uuid": "dae3f484-9f24-4f24-96c3-253465d0d369", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 96636764160, 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "btime": "2024-09-15T03:40:09.388250+0000", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "main", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_blocks": "23592960", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_blocks_per_key": "128", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_bytes_per_block": "4096", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_size": "96636764160", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs": "1", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_fsid": "da8e668f-207e-4717-9e66-7070b1986601", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:41:27.089 INFO:teuthology.orchestra.run.smithi105.stdout: "created_at": "2024-09-15T03:40:11.732295Z", 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: "kv_backend": "rocksdb", 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: "magic": "ceph osd volume v026", 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: "mkfs_done": "yes", 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: "ready": "ready", 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: "require_osd_release": "17", 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": "3" 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:27.090 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:27.092 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.3 2024-09-15T03:41:27.092 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 show-sharding 2024-09-15T03:41:27.097 INFO:tasks.rados.rados.0.smithi105.stdout:170: finishing write tid 1 to smithi10536496-15 2024-09-15T03:41:27.099 INFO:tasks.rados.rados.0.smithi105.stdout:170: finishing write tid 2 to smithi10536496-15 2024-09-15T03:41:27.099 INFO:tasks.rados.rados.0.smithi105.stdout:170: finishing write tid 3 to smithi10536496-15 2024-09-15T03:41:27.101 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.079699538, 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008773889, 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019951+0000", 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7080000000000002e-05 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020259+0000", 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030882000000000001 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020269+0000", 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4690000000000003e-06 2024-09-15T03:41:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020353+0000", 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4121000000000003e-05 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020367+0000", 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.401e-05 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028603+0000", 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082364340000000008 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028667+0000", 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3665999999999997e-05 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.078852261000002, 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075521720000000002, 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020741+0000", 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020828+0000", 2024-09-15T03:41:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7108999999999996e-05 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020883+0000", 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4818000000000003e-05 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020888+0000", 2024-09-15T03:41:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.763e-06 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020921+0000", 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2780000000000001e-05 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020926+0000", 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8359999999999997e-06 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028240+0000", 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073138810000000004 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028293+0000", 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2281000000000003e-05 2024-09-15T03:41:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.078760411000001, 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075072999999999997, 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020833+0000", 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4400000000000001e-07 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020843+0000", 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020899+0000", 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6416000000000003e-05 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020905+0000", 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4670000000000003e-06 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022364+0000", 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001459606 2024-09-15T03:41:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022373+0000", 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5229999999999995e-06 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028305+0000", 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059320400000000004 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028340+0000", 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4603000000000001e-05 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.075122123, 2024-09-15T03:41:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072410060000000004, 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.74e-07 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024570+0000", 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9729999999999996e-05 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024599+0000", 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8530999999999998e-05 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024605+0000", 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2280000000000003e-06 2024-09-15T03:41:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024648+0000", 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2466000000000003e-05 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024656+0000", 2024-09-15T03:41:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8690000000000004e-06 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031688+0000", 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070326130000000001 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031712+0000", 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3394999999999999e-05 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=4, reqid=client.4562.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.074177886000001, 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063223339999999998, 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025416+0000", 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4600000000000004e-07 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025430+0000", 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4636e-05 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025456+0000", 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5718e-05 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025462+0000", 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025538+0000", 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5916999999999998e-05 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025548+0000", 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9620000000000005e-06 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031716+0000", 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061672910000000001 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031737+0000", 2024-09-15T03:41:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1569999999999998e-05 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.073877635999999, 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061423019999999997, 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025716+0000", 2024-09-15T03:41:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5199999999999996e-07 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025913+0000", 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000197252 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025939+0000", 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.552e-05 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025944+0000", 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7320000000000003e-06 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026100+0000", 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575499999999999 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026109+0000", 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3700000000000001e-06 2024-09-15T03:41:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031831+0000", 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005721271 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031857+0000", 2024-09-15T03:41:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.675e-05 2024-09-15T03:41:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.063221572, 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059101470000000001, 2024-09-15T03:41:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036372+0000", 2024-09-15T03:41:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036625+0000", 2024-09-15T03:41:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025348599999999998 2024-09-15T03:41:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036653+0000", 2024-09-15T03:41:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8116e-05 2024-09-15T03:41:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7190000000000007e-06 2024-09-15T03:41:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036702+0000", 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0129000000000002e-05 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036711+0000", 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042262+0000", 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005551177 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042281+0000", 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9004e-05 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.074971862, 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067171740000000002, 2024-09-15T03:41:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024622+0000", 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1489999999999999e-06 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024960+0000", 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000337685 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024999+0000", 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9569e-05 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025010+0000", 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0417e-05 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025062+0000", 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2043999999999998e-05 2024-09-15T03:41:27.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025074+0000", 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2227e-05 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031307+0000", 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062330859999999997 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031338+0000", 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0997000000000002e-05 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.073581956, 2024-09-15T03:41:27.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079502919999999994, 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:27.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026099+0000", 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7769999999999995e-05 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026141+0000", 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2227000000000001e-05 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026149+0000", 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3730000000000004e-06 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026244+0000", 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5662999999999995e-05 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026255+0000", 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0603e-05 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033934+0000", 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076787699999999997 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033961+0000", 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7206999999999999e-05 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.072138800999999, 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065285430000000004, 2024-09-15T03:41:27.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027455+0000", 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9299999999999996e-07 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027493+0000", 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7716000000000003e-05 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027532+0000", 2024-09-15T03:41:27.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9904000000000003e-05 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027539+0000", 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027595+0000", 2024-09-15T03:41:27.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5841000000000003e-05 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027605+0000", 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7399999999999999e-06 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033967+0000", 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063620969999999997 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033982+0000", 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5389000000000001e-05 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=64, reqid=client.4562.0:284, at_version=0'0, trim_to=0'0, roll_forward_to=45'28))", 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:27.158 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.060977447000001, 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090921019999999995, 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038616+0000", 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038629+0000", 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2891000000000001e-05 2024-09-15T03:41:27.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038669+0000", 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9725000000000003e-05 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038682+0000", 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3095000000000001e-05 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038746+0000", 2024-09-15T03:41:27.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3850999999999996e-05 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038758+0000", 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2329e-05 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047686+0000", 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089280980000000006 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047707+0000", 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1089000000000001e-05 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:300 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~442368] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.090349611000001, 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0389183930000001, 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009245+0000", 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.556e-06 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009830+0000", 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058575800000000003 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009844+0000", 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3247e-05 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009851+0000", 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4850000000000003e-06 2024-09-15T03:41:27.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010438+0000", 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058667099999999998 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010441+0000", 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2119999999999999e-06 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031346+0000", 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0209051899999999 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031477+0000", 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130712 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.032397+0000", 2024-09-15T03:41:27.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091970699999999997 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037882+0000", 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054851639999999998 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048077+0000", 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010194896 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048161+0000", 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4795000000000004e-05 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:301 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~720896] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.089725028, 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456429519999999, 2024-09-15T03:41:27.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 301 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010594+0000", 2024-09-15T03:41:27.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072566600000000003 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010605+0000", 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1228e-05 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010611+0000", 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8390000000000003e-06 2024-09-15T03:41:27.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010679+0000", 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7966999999999995e-05 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010681+0000", 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1409999999999999e-06 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033259+0000", 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.022577372 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033385+0000", 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012635200000000001 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.034673+0000", 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001287853 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038109+0000", 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003436508 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055462+0000", 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017353200999999999 2024-09-15T03:41:27.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055511+0000", 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8217999999999998e-05 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.177 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:302 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~49152] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.088974647000001, 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.045100688, 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.179 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 302 2024-09-15T03:41:27.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:27.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010619+0000", 2024-09-15T03:41:27.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0699999999999996e-07 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010663+0000", 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552000000000003e-05 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010680+0000", 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6175999999999999e-05 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010685+0000", 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8619999999999999e-06 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010726+0000", 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0259000000000002e-05 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010727+0000", 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.753e-06 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035776+0000", 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0250488879999999 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035909+0000", 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013254900000000001 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.036628+0000", 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00071956899999999998 2024-09-15T03:41:27.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038283+0000", 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016543490000000001 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055650+0000", 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017367301000000002 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055719+0000", 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8622999999999997e-05 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:303 3.11s0 3:8f673fa9:::smithi10536496-24:head [setxattr _header (56) in=7b] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.088896761000001, 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0452317200000001, 2024-09-15T03:41:27.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 303 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010697+0000", 2024-09-15T03:41:27.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010700+0000", 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2509999999999999e-06 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010706+0000", 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4989999999999999e-06 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010711+0000", 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2250000000000002e-06 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010765+0000", 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4784000000000003e-05 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010767+0000", 2024-09-15T03:41:27.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4160000000000001e-06 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037038+0000", 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.026271521 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037181+0000", 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014324800000000001 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037675+0000", 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049306399999999998 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.039474+0000", 2024-09-15T03:41:27.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017996729999999999 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055837+0000", 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016362393999999999 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055928+0000", 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1051000000000006e-05 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:304 3.11s0 3:8f673fa9:::smithi10536496-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.088877839, 2024-09-15T03:41:27.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.050398215, 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 304 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-09-15T03:41:27.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010717+0000", 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.125e-06 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010724+0000", 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6880000000000002e-06 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010728+0000", 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.777e-06 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010795+0000", 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7277e-05 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010796+0000", 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037918+0000", 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.02712114 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038025+0000", 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010774 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056315+0000", 2024-09-15T03:41:27.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018289716000000001 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056417+0000", 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102336 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061113+0000", 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046956749999999998 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:317 3.11s0 3:8b799747:::smithi10536496-30:head [rollback e] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.040512365, 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010406076, 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 317 2024-09-15T03:41:27.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059081+0000", 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.151e-06 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:41:27.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059096+0000", 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4499999999999997e-06 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060096+0000", 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010001019999999999 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060189+0000", 2024-09-15T03:41:27.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2320999999999999e-05 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060639+0000", 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044999599999999998 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062024+0000", 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013859650000000001 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069431+0000", 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074061190000000001 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069486+0000", 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5831999999999997e-05 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:318 3.11s0 3:8b799747:::smithi10536496-30:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.040491329, 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010504207999999999, 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 318 2024-09-15T03:41:27.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059102+0000", 2024-09-15T03:41:27.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6600000000000002e-07 2024-09-15T03:41:27.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059104+0000", 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.215e-06 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059110+0000", 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0279999999999997e-06 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059114+0000", 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3340000000000003e-06 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061223+0000", 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002108249 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061279+0000", 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6418000000000001e-05 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061697+0000", 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000418109 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062208+0000", 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051137000000000003 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069565+0000", 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073569799999999999 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069606+0000", 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0139000000000003e-05 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:374 3.11s0 3:8b799747:::smithi10536496-30:head [delete] snapc e=[c,b] ondisk+write+known_if_redirected+full_force+supports_pool_eio e51)", 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.0691849609999995, 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086570359999999999, 2024-09-15T03:41:27.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 374 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030410+0000", 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1500000000000002e-06 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030418+0000", 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5989999999999996e-06 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030442+0000", 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4612000000000001e-05 2024-09-15T03:41:27.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030450+0000", 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2959999999999999e-06 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030513+0000", 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030644+0000", 2024-09-15T03:41:27.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130591 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.031027+0000", 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038316199999999999 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.031678+0000", 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00065064699999999997 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.039008+0000", 2024-09-15T03:41:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073303159999999999 2024-09-15T03:41:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.039065+0000", 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6904000000000003e-05 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 56/51 ECSubWrite(tid=35, reqid=osd.7.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=56'16))", 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:23.072980+0000", 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.0266131349999998, 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064563820000000001, 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072980+0000", 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072980+0000", 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072981+0000", 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.64e-07 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072993+0000", 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2063e-05 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073043+0000", 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0621999999999997e-05 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073050+0000", 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1080000000000004e-06 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073116+0000", 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5690000000000003e-05 2024-09-15T03:41:27.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073129+0000", 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2707999999999999e-05 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.079401+0000", 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062725139999999999 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.079436+0000", 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4712999999999998e-05 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.227 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:27.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:27.229 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:27.229 INFO:tasks.rados.rados.0.smithi105.stdout:170: finishing write tid 4 to smithi10536496-15 2024-09-15T03:41:27.229 INFO:tasks.rados.rados.0.smithi105.stdout:170: finishing write tid 5 to smithi10536496-15 2024-09-15T03:41:27.229 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 12 (ObjNum 86 snap 13 seq_num 86) dirty exists 2024-09-15T03:41:27.229 INFO:tasks.rados.rados.0.smithi105.stdout:170: left oid 15 (ObjNum 86 snap 13 seq_num 86) 2024-09-15T03:41:27.230 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:27.231 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:27.231 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:27.228+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:27.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:27.330 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:27.329+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:27.367 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:27.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:27.367 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:27.367 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:27.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:27.431 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:27.430+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:27.472 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:27.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:27.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:27.532 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:27.531+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:15 3.0s0 3:092ceaee:::smithi10536496-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.559408348000002, 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021232414000000002, 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018672+0000", 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5680000000000001e-06 2024-09-15T03:41:27.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018678+0000", 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1760000000000003e-06 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018680+0000", 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.243e-06 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020924+0000", 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022433499999999999 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020978+0000", 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4500000000000003e-05 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032096+0000", 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011118304000000001 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032182+0000", 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5785000000000006e-05 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039904+0000", 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077221329999999999 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:27 3.0s0 3:0304afdb:::smithi10536496-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.558949949999999, 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028611145000000001, 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 27 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019131+0000", 2024-09-15T03:41:27.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019132+0000", 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7460000000000001e-06 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019139+0000", 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019143+0000", 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6720000000000003e-06 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023415+0000", 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042719689999999996 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023491+0000", 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5638999999999999e-05 2024-09-15T03:41:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034815+0000", 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011324555 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034893+0000", 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7682999999999995e-05 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047741+0000", 2024-09-15T03:41:27.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01284832 2024-09-15T03:41:27.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.591 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.556096753999999, 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.592 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:27.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.549489507000001, 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:41:27.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.603 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.549471893, 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.604 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:41:27.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.544746688, 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:41:27.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:41:27.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:64 3.0s0 3:0215fa29:::smithi10536496-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.532558108, 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.023494108999999999, 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 64 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045523+0000", 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.358e-06 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045894+0000", 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037004799999999999 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045904+0000", 2024-09-15T03:41:27.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456e-05 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045909+0000", 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6029999999999998e-06 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045938+0000", 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9054e-05 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046014+0000", 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6334000000000005e-05 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046523+0000", 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000508903 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050383+0000", 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003860086 2024-09-15T03:41:27.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068971+0000", 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018587939000000001 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.069016+0000", 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5327999999999997e-05 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.532164944000002, 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:41:27.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:27.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:41:27.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.532153401999999, 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:41:27.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:41:27.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:41:27.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:41:27.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:27.632 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.512961775000001, 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:27.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:41:27.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.512477879999999, 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.638 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:27.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:41:27.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.512436236999999, 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:41:27.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:41:27.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:41:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:41:27.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.510182790000002, 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:41:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:41:27.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:27.654 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.509194448999999, 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.452814099000001, 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036705979, 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125267+0000", 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125270+0000", 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161517+0000", 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036246426999999998 2024-09-15T03:41:27.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161523+0000", 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161562+0000", 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8571000000000001e-05 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161972+0000", 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041002299999999997 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.416712, 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:41:27.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:41:27.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.412744335999999, 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:27.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:41:27.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:287 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 8] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.549800747000001, 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.820123202, 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 287 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.672 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028281+0000", 2024-09-15T03:41:27.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5200000000000001e-06 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028286+0000", 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028300+0000", 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3668e-05 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028307+0000", 2024-09-15T03:41:27.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9539999999999998e-06 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029951+0000", 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001644419 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029955+0000", 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6260000000000001e-06 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037676+0000", 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077205920000000001 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037910+0000", 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234212 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057184+0000", 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019273795999999999 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.059033+0000", 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00184902 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848316+0000", 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78928352999999996 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848403+0000", 2024-09-15T03:41:27.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6583999999999996e-05 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:288 3.10s0 3:092ceaee:::smithi10536496-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.549763806, 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030561617999999999, 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:27.678 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 288 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028317+0000", 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299999999999996e-07 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028320+0000", 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0019999999999998e-06 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028328+0000", 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9379999999999992e-06 2024-09-15T03:41:27.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028357+0000", 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8770999999999999e-05 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030005+0000", 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016479140000000001 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:27.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030007+0000", 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5229999999999998e-06 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038132+0000", 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081245950000000001 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038195+0000", 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2838999999999999e-05 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058661+0000", 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020466254999999999 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058771+0000", 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010993999999999999 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058878+0000", 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107268 2024-09-15T03:41:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 45/44 ECSubWriteReply(tid=39, last_complete=45'16, committed=1, applied=1))", 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.513129387999999, 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78386887299999997, 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:41:27.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064956+0000", 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848087+0000", 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78313173400000002 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848099+0000", 2024-09-15T03:41:27.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1173e-05 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848179+0000", 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0921999999999995e-05 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848820+0000", 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064024899999999996 2024-09-15T03:41:27.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:27.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:27.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:27.689 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:27.690 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:27.688+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:27.789 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:27.789 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:27.789 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:27.790 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:27.790 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:27.789+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:27.797 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:27.891 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:27.889+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:27.894 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:27.894 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:27.894 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:27.894 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:27.894 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:27.899 INFO:teuthology.orchestra.run.smithi105.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P 2024-09-15T03:41:27.903 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:27.904 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.3 2024-09-15T03:41:27.904 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-15T03:41:27.904 INFO:tasks.thrashosds.thrasher:applying new sharding: P(1,0-30) L(3,8-24) 2024-09-15T03:41:27.904 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 --sharding 'P(1,0-30) L(3,8-24)' reshard 2024-09-15T03:41:27.992 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:27.991+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:27.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:41:27.992+0000 7f91bd209780 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:41:28.045 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:28.045 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:28.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:28.045 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.010688646999998, 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0090780140000000006, 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:28.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0100000000000005e-07 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5333e-05 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041513+0000", 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084585600000000004 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041517+0000", 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.318e-06 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041556+0000", 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8426000000000001e-05 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041563+0000", 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6930000000000007e-06 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042090+0000", 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052617600000000005 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042117+0000", 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6906999999999998e-05 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.050 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.996752488999999, 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:41:28.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:41:28.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:41:28.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:41:28.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:41:28.054 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.985123590000001, 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:41:28.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:41:28.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.060 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:91 3.ds0 3:baae61d4:::smithi10536496-31:head [append 0~507904,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.967904180000001, 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0073884809999999997, 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 91 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:28.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075823+0000", 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.075824+0000", 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3069999999999999e-06 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076181+0000", 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00035651600000000003 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076196+0000", 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5308e-05 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076206+0000", 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4469999999999995e-06 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076297+0000", 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1287999999999997e-05 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076465+0000", 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016844599999999999 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.077478+0000", 2024-09-15T03:41:28.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010127140000000001 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080498+0000", 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030201759999999999 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083146+0000", 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026477100000000002 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083212+0000", 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5568999999999995e-05 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:92 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.967505211999999, 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078039399999999997, 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.066 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 92 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:28.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076223+0000", 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076226+0000", 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6210000000000001e-06 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076234+0000", 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2740000000000002e-06 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076238+0000", 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4329999999999997e-06 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078113+0000", 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018749769999999999 2024-09-15T03:41:28.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078219+0000", 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106016 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078626+0000", 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040713400000000002 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080519+0000", 2024-09-15T03:41:28.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001892712 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083957+0000", 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034382610000000002 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084026+0000", 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8678999999999998e-05 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.071 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.967480089999999, 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.072 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:41:28.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:41:28.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.957654341000001, 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:41:28.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:41:28.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.080 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.928577308000001, 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.081 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:28.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:41:28.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.917660824999999, 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017309390000000001, 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126067+0000", 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:41:28.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126072+0000", 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3920000000000002e-06 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143252+0000", 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017179104000000001 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143257+0000", 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143315+0000", 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7958000000000003e-05 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143376+0000", 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0535000000000002e-05 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.089 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=19, reqid=client.4562.0:144, at_version=35'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.897275671999999, 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015149222, 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:28.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.85e-07 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146470+0000", 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7309999999999999e-05 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160975+0000", 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014505123 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160981+0000", 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161035+0000", 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3517999999999998e-05 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.065e-05 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161571+0000", 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052504699999999995 2024-09-15T03:41:28.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161601+0000", 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=23, reqid=client.4562.0:154, at_version=35'12, trim_to=0'0, roll_forward_to=35'11))", 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.878348275, 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.083287300999999994, 2024-09-15T03:41:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165380+0000", 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0999999999999997e-07 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165457+0000", 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6823999999999995e-05 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247921+0000", 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082464198000000002 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247926+0000", 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9880000000000004e-06 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247954+0000", 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8306e-05 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247960+0000", 2024-09-15T03:41:28.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7570000000000001e-06 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248650+0000", 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000689816 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248666+0000", 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6501999999999999e-05 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=21, reqid=client.4562.0:290, at_version=45'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.013455995000001, 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077833549999999996, 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030272+0000", 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2399999999999997e-07 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030290+0000", 2024-09-15T03:41:28.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7683999999999999e-05 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030328+0000", 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8362999999999998e-05 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030335+0000", 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030374+0000", 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8415999999999999e-05 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030380+0000", 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7649999999999996e-06 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038032+0000", 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076514269999999997 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038055+0000", 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2776e-05 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s1 3.1f172733 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.032863184, 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017692614, 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.103 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010866+0000", 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010929+0000", 2024-09-15T03:41:28.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010944+0000", 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4701e-05 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010952+0000", 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4610000000000002e-06 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011020+0000", 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8553000000000006e-05 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011023+0000", 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8150000000000002e-06 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028536+0000", 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017512913 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028557+0000", 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0716e-05 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:310 3.as1 3:5629793d:::smithi10536496-26:head [delete] snapc e=[e,c,b] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.038268065, 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0408854540000001, 2024-09-15T03:41:28.108 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 310 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005461+0000", 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.643e-06 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005465+0000", 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:41:28.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005480+0000", 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5407999999999999e-05 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005487+0000", 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4229999999999999e-06 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008383+0000", 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002895309 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008386+0000", 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8949999999999998e-06 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.031743+0000", 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0233571130000001 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.032058+0000", 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000315123 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.033215+0000", 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011571579999999999 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.037843+0000", 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046277380000000002 2024-09-15T03:41:28.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.045957+0000", 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081145560000000002 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046345+0000", 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000387229 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 50/49 ECSubWrite(tid=151, reqid=client.4562.0:335, at_version=50'20, trim_to=0'0, roll_forward_to=50'18))", 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.912956006, 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067600538000000002, 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130773+0000", 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130954+0000", 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018089000000000001 2024-09-15T03:41:28.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196934+0000", 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065979530999999994 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196943+0000", 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197012+0000", 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8745999999999995e-05 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197025+0000", 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2909e-05 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198352+0000", 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001327188 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198372+0000", 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9933999999999999e-05 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:347 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.895456209000001, 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089938310000000007, 2024-09-15T03:41:28.119 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 347 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.58e-07 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148275+0000", 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0269999999999997e-06 2024-09-15T03:41:28.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148282+0000", 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.195e-06 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148286+0000", 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3669999999999999e-06 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.152643+0000", 2024-09-15T03:41:28.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043572790000000004 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.152776+0000", 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000133156 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.153340+0000", 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00056305400000000001 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154283+0000", 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00094319800000000002 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.156961+0000", 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026778100000000001 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157265+0000", 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030428699999999998 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.124 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:348 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.895436606000001, 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010390460000000001, 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 348 2024-09-15T03:41:28.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.84e-07 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148293+0000", 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.858e-06 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148299+0000", 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9780000000000002e-06 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148303+0000", 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8249999999999998e-06 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154925+0000", 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066227459999999997 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.155002+0000", 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6196e-05 2024-09-15T03:41:28.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157677+0000", 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026754650000000001 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157794+0000", 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117167 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.158681+0000", 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088684099999999997 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.18s5 51/49 ECSubWrite(tid=153, reqid=client.4562.0:369, at_version=51'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.008665023000001, 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007414889, 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.130 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035063+0000", 2024-09-15T03:41:28.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035480+0000", 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041686700000000002 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035525+0000", 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5948999999999997e-05 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035532+0000", 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1349999999999997e-06 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035566+0000", 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4785e-05 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035574+0000", 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.042460+0000", 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068862660000000003 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.042477+0000", 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 56/44 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=56'30, trim_to=0'0, roll_forward_to=51'28))", 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.9698425950000003, 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096114640000000001, 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073915+0000", 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9552000000000001e-05 2024-09-15T03:41:28.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073949+0000", 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4022000000000001e-05 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073955+0000", 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1349999999999997e-06 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074002+0000", 2024-09-15T03:41:28.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7089000000000003e-05 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074013+0000", 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0446e-05 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083471+0000", 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094584449999999994 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083496+0000", 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4964e-05 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 56/44 ECSubWrite(tid=41, reqid=osd.15.0:41, at_version=56'32, trim_to=0'0, roll_forward_to=51'28))", 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:23.074553+0000", 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.969173777, 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092758249999999997, 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.139 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074553+0000", 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074553+0000", 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074554+0000", 2024-09-15T03:41:28.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4799999999999997e-07 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074595+0000", 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1145e-05 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074631+0000", 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5735999999999998e-05 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074638+0000", 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5100000000000004e-06 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.075409+0000", 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00077193300000000004 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.075419+0000", 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9159999999999996e-06 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083796+0000", 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0083766959999999994 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083829+0000", 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3141e-05 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.143 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:28.145 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:28.146 INFO:tasks.rados.rados.0.smithi105.stdout:170: done (2 left) 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:171: done (1 left) 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:172: done (0 left) 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:173: copy_from oid 19 from oid 35 current snap is 14 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 15 current snap is 14 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:174: seq_num 87 ranges {1703936=704512,2408448=688128,3096576=81920} 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:174: writing smithi10536496-15 from 1703936 to 2408448 tid 1 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:174: writing smithi10536496-15 from 2408448 to 3096576 tid 2 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:174: writing smithi10536496-15 from 3096576 to 3178496 tid 3 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:175: rollback oid 4 current snap is 14 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 4 to 12 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:173: finishing copy_from racing read to smithi10536496-19 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:176: delete oid 7 current snap is 14 2024-09-15T03:41:28.147 INFO:tasks.rados.rados.0.smithi105.stdout:173: finishing copy_from to smithi10536496-19 2024-09-15T03:41:28.148 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:28.148 INFO:tasks.rados.rados.0.smithi105.stdout:175: finishing rollback tid 0 to smithi10536496-4 2024-09-15T03:41:28.148 INFO:tasks.rados.rados.0.smithi105.stdout:174: finishing write tid 1 to smithi10536496-15 2024-09-15T03:41:28.148 INFO:tasks.rados.rados.0.smithi105.stdout:173: done (3 left) 2024-09-15T03:41:28.148 INFO:tasks.rados.rados.0.smithi105.stdout:176: done (2 left) 2024-09-15T03:41:28.148 INFO:tasks.rados.rados.0.smithi105.stdout:177: snap_remove snap 8 2024-09-15T03:41:28.148 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:28.143+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:28.245 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:28.244+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:28.288 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:28.289 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:28.289 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:28.289 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:28.298 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:28.311 INFO:teuthology.orchestra.run.smithi105.stdout:reshard success 2024-09-15T03:41:28.346 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:28.345+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:28.394 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:28.395 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:28.402 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:28.447 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:28.446+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:30 3.fs0 3:fff44eab:::smithi10536496-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.520910285999999, 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062015339000000003, 2024-09-15T03:41:28.542 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 30 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019185+0000", 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999995e-07 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019188+0000", 2024-09-15T03:41:28.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6160000000000002e-06 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019196+0000", 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8699999999999992e-06 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019201+0000", 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8960000000000002e-06 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019234+0000", 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3402000000000003e-05 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.019239+0000", 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1450000000000001e-06 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025625+0000", 2024-09-15T03:41:28.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063864280000000004 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.025701+0000", 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6044999999999994e-05 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033568+0000", 2024-09-15T03:41:28.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078674130000000002 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033650+0000", 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1404999999999997e-05 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.081199+0000", 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047549438999999999 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.548 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:53 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.504420874000001, 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.089580073999999996, 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 53 2024-09-15T03:41:28.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035674+0000", 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7000000000000005e-07 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035677+0000", 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4589999999999998e-06 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035684+0000", 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2659999999999998e-06 2024-09-15T03:41:28.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035694+0000", 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8849999999999994e-06 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096889100000000004 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036718+0000", 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5436000000000003e-05 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037156+0000", 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043800399999999998 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037610+0000", 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00045398200000000002 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125179+0000", 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.087569006000000005 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125253+0000", 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4574999999999997e-05 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:54 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.50439454, 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093566412000000002, 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 54 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035702+0000", 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1490000000000001e-06 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035709+0000", 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6630000000000004e-06 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035713+0000", 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7780000000000001e-06 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037308+0000", 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015947769999999999 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037380+0000", 2024-09-15T03:41:28.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2293999999999996e-05 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125568+0000", 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.088188213000000001 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125642+0000", 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4215999999999996e-05 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.129266+0000", 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003623799 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:119 3.fs0 3:f17f8228:::smithi10536496-40:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.433654527000002, 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046131245000000001, 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.560 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 119 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106440+0000", 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2e-07 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106442+0000", 2024-09-15T03:41:28.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0150000000000002e-06 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106446+0000", 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6509999999999999e-06 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106448+0000", 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.21e-06 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107892+0000", 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001444435 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107983+0000", 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0179999999999994e-05 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108344+0000", 2024-09-15T03:41:28.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036168399999999997 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.110942+0000", 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002597495 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152509+0000", 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041567518999999997 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152571+0000", 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1636000000000004e-05 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:120 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.433627391000002, 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453923999999997, 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.566 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 120 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106467+0000", 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0400000000000002e-07 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106469+0000", 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.519e-06 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106474+0000", 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5090000000000001e-06 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.106480+0000", 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.285e-06 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108518+0000", 2024-09-15T03:41:28.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020372820000000001 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.108590+0000", 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2280999999999995e-05 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152611+0000", 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044021429000000001 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.152655+0000", 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3671000000000003e-05 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.156921+0000", 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0042656439999999999 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:171 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.571 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.346108165, 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62108922, 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 171 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193987+0000", 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193989+0000", 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6410000000000002e-06 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193994+0000", 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193996+0000", 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:41:28.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00136275 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195404+0000", 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5433999999999999e-05 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195711+0000", 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030663100000000003 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196156+0000", 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044516899999999998 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815021+0000", 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61886439199999999 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815075+0000", 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4685000000000002e-05 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:172 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.346076851999999, 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62217043699999997, 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.577 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 172 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194018+0000", 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5699999999999998e-07 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194020+0000", 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194026+0000", 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194028+0000", 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.475e-06 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195860+0000", 2024-09-15T03:41:28.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018314609999999999 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195941+0000", 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1464999999999993e-05 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815378+0000", 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61943704499999996 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815470+0000", 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1351e-05 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.816188+0000", 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071828500000000004 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:175 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:28.582 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.346030945999999, 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068787259000000003, 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 175 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:28.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194064+0000", 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194065+0000", 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5540000000000001e-06 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194068+0000", 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0730000000000001e-06 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194071+0000", 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4770000000000002e-06 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195541+0000", 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014705670000000001 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195649+0000", 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108057 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.262851+0000", 2024-09-15T03:41:28.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067201307000000002 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 35/32 ECSubWriteReply(tid=30, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.343494624000002, 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61874190500000004, 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:41:28.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196603+0000", 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7719999999999999e-06 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814918+0000", 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61831525899999995 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814925+0000", 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1629999999999999e-06 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814972+0000", 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7231999999999998e-05 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815342+0000", 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000369019 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:300 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.535457453999999, 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0215413659999999, 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 300 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004638+0000", 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6020000000000001e-06 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004976+0000", 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033792499999999999 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004994+0000", 2024-09-15T03:41:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7507e-05 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.005001+0000", 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008081+0000", 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003079675 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008085+0000", 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3420000000000001e-06 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026131+0000", 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0180457119999999 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026178+0000", 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6953000000000003e-05 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:301 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.530943195000001, 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0170563379999999, 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 301 2024-09-15T03:41:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009153+0000", 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4569999999999999e-06 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009921+0000", 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076796899999999997 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009955+0000", 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3696999999999999e-05 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009971+0000", 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5441e-05 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010025+0000", 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4712000000000001e-05 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010029+0000", 2024-09-15T03:41:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.63e-06 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026201+0000", 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016171661 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026207+0000", 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7710000000000001e-06 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:302 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.5301077, 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016248987, 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 302 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7700000000000004e-07 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010036+0000", 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8857999999999999e-05 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010045+0000", 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6200000000000005e-06 2024-09-15T03:41:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010049+0000", 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3649999999999997e-06 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010226+0000", 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017731600000000001 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010228+0000", 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3090000000000001e-06 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026231+0000", 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016003078 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026236+0000", 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6639999999999997e-06 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:303 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.530037344, 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016202901, 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 303 2024-09-15T03:41:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1399999999999997e-07 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010061+0000", 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010068+0000", 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9779999999999999e-06 2024-09-15T03:41:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010072+0000", 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0489999999999997e-06 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010289+0000", 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021628399999999999 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010290+0000", 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.201e-06 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026255+0000", 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015965209 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026260+0000", 2024-09-15T03:41:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6659999999999999e-06 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:304 3.11s1 3.95fce6f1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.530018491, 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0162086930000001, 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 304 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010078+0000", 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.711e-06 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010082+0000", 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1320000000000004e-06 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010084+0000", 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5450000000000002e-06 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010311+0000", 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022657600000000001 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010314+0000", 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.565e-06 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026281+0000", 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015966967 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026284+0000", 2024-09-15T03:41:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 47/32 ECSubWriteReply(tid=66, last_complete=47'18, committed=1, applied=1))", 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.52746125, 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16187408, 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3799999999999999e-07 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012636+0000", 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.345e-06 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173941+0000", 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16130529900000001 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173963+0000", 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.243e-05 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174016+0000", 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2948000000000002e-05 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174507+0000", 2024-09-15T03:41:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049081999999999999 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s0 3:cce4e8f8:::smithi10536496-4:head [create,setxattr Yt`Ot\\d[ZfNaQaqU[W\\PSa^pNWWUkki`hjlaoWcR (805) in=40b,setxattr ca[gmd`SV\\jp[aXclsea`kka[hPURW\\j[hao]pna (594) in=40b,setxattr jjQP\\pmOqcO\\iaNd[l\\^a]fXSRsgi\\sVYrQS[]ia (1614) in=40b,setxattr VeZgjt^aNniSnQhWPO[qm\\[itg[XR[[jXsbagns_ (454) in=40b,setxattr jath]ttaVem][bRrjP]pdOZWWo__SYO]VOaZhXRj (1858) in=40b,setxattr _oSdRYPilSVobVgWRbrP_piatjWaoYQ\\qcaVbh^g (138) in=40b,setxattr bTSZgV_k`ccTPieZXaVqe[OXkZefRqrsPPlisa^a (946) in=40b,setxattr OkXYkpcX_kP[TmZOosPhjlctOQRUagSbp`XaYrXW (910) in=40b,setxattr rjkSWecoRfpnaXdQmX`a`dPYt]P[_hRUfj`SdY[ (911) in=39b,setxattr OcknbT^]PQTVTj\\iodYl`oq[]Te]Us]e[nbQYY^N (1330) in=40b,setxattr r^qtXit^UbRgSb`o\\aoaSSnZqmZeWhhehTPZsgUO (418) in=40b,setxattr bnoZ`pV]YYNYqogeSaPofWR_PtahQZ_fR_h_nTnT (1190) in=40b,setxattr rjred]_aNiaaolkeaffPXTfZRd[[_PYZaUXNOUUt (33) in=40b,setxattr gnal\\XR[qfmfOecOmacYb_WWa\\W[rRPNWne`tlNT (1437) in=40b,setxattr sZmfXOTVQZ^VacNNaltSYOghkYNpdSfdZRhnXTn_ (4981) in=40b,setxattr TsTNdOVOpOPdoXYcpnamYadP^epnqllcbVsWNp^[ (17753) in=40b,setxattr PXkZNpT]nPaTrW]j`o_[RWiQrOrco]e_foQTXmO_ (1997) in=40b,setxattr sX[hspQmOVmae`epqt]pac_ZNQVn`YjUVU]QfSg^ (1051) in=40b,setxattr ^gV_afUghN\\gRs]pcaefnp[badg_a`]ROfd_ZZnW (5752) in=40b,setxattr d\\kRaqs`ZmPOe`_`ZimVtjOoQ[NieeSSZocTVNl_ (313) in=40b] snapc c=[c,b] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.528152982, 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0346458860000001, 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:41:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011942+0000", 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2100000000000002e-07 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011959+0000", 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7248000000000002e-05 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011965+0000", 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8880000000000002e-06 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011969+0000", 2024-09-15T03:41:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8929999999999998e-06 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.012000+0000", 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1004999999999999e-05 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.012002+0000", 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.289e-06 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.035078+0000", 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0230762010000001 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.035294+0000", 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000215625 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.036366+0000", 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00107211 2024-09-15T03:41:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046017+0000", 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096507329999999999 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046134+0000", 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011691199999999999 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046587+0000", 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000453561 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:321 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.456933412, 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11431013299999999, 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 321 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083167+0000", 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5169999999999999e-06 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083174+0000", 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9890000000000003e-06 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083178+0000", 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2220000000000004e-06 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.084760+0000", 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001582008 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.084856+0000", 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5461e-05 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085326+0000", 2024-09-15T03:41:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046985000000000001 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.087826+0000", 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025000759999999999 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197378+0000", 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.109552474 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197471+0000", 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2888999999999998e-05 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:322 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.456908879, 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12070206999999999, 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 322 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083186+0000", 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.89e-07 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083188+0000", 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1009999999999999e-06 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083195+0000", 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083199+0000", 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.41e-06 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085640+0000", 2024-09-15T03:41:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002441838 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085722+0000", 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1780999999999995e-05 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197721+0000", 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.111998507 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197792+0000", 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0963000000000005e-05 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.203887+0000", 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060957049999999999 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 50/32 ECSubWriteReply(tid=75, last_complete=50'22, committed=1, applied=1))", 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.453130593999999, 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.110727595, 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086965+0000", 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6109999999999999e-06 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197067+0000", 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.110101613 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197072+0000", 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7940000000000002e-06 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197329+0000", 2024-09-15T03:41:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025732199999999999 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197691+0000", 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036200000000000002 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 50/32 ECSubWriteReply(tid=80, last_complete=50'22, committed=1, applied=1))", 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.339489312, 2024-09-15T03:41:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.82440747800000003, 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.91e-07 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200607+0000", 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024931+0000", 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.82432465200000005 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024938+0000", 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2129999999999996e-06 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024985+0000", 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7713999999999998e-05 2024-09-15T03:41:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.025012+0000", 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7058000000000001e-05 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:28.646 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:28.648 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:28.648 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:28.647+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:28.746 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:28.746 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:28.748 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:28.746+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:28.752 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.3 2024-09-15T03:41:28.753 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-3 fsck 2024-09-15T03:41:28.754 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:28.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-09-15T03:41:28.799+0000 7fec27adb780 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:41:28.849 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:28.848+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:28.892 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:28.893 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:28.893 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:28.893 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:28.893 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:28.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:28.950 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:28.949+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:29.005 INFO:tasks.rados.rados.0.smithi105.stdout:177: done (2 left) 2024-09-15T03:41:29.005 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 44 current snap is 14 2024-09-15T03:41:29.005 INFO:tasks.rados.rados.0.smithi105.stdout:178: seq_num 89 ranges {1720320=737280,2457600=573440,3031040=196608} 2024-09-15T03:41:29.020 INFO:tasks.rados.rados.0.smithi105.stdout:178: writing smithi10536496-44 from 1720320 to 2457600 tid 1 2024-09-15T03:41:29.024 INFO:tasks.rados.rados.0.smithi105.stdout:178: writing smithi10536496-44 from 2457600 to 3031040 tid 2 2024-09-15T03:41:29.026 INFO:tasks.rados.rados.0.smithi105.stdout:178: writing smithi10536496-44 from 3031040 to 3227648 tid 3 2024-09-15T03:41:29.026 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 36 current snap is 14 2024-09-15T03:41:29.026 INFO:tasks.rados.rados.0.smithi105.stdout:175: finishing rollback tid 1 to smithi10536496-4 2024-09-15T03:41:29.027 INFO:tasks.rados.rados.0.smithi105.stdout:175: finishing rollback tid 2 to smithi10536496-4 2024-09-15T03:41:29.027 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 33 (ObjNum 85 snap 12 seq_num 85) dirty exists 2024-09-15T03:41:29.027 INFO:tasks.rados.rados.0.smithi105.stdout:174: finishing write tid 2 to smithi10536496-15 2024-09-15T03:41:29.027 INFO:tasks.rados.rados.0.smithi105.stdout:179: seq_num 90 ranges {245760=753664,999424=720896,1720320=49152} 2024-09-15T03:41:29.033 INFO:tasks.rados.rados.0.smithi105.stdout:179: writing smithi10536496-36 from 245760 to 999424 tid 1 2024-09-15T03:41:29.038 INFO:tasks.rados.rados.0.smithi105.stdout:179: writing smithi10536496-36 from 999424 to 1720320 tid 2 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:179: writing smithi10536496-36 from 1720320 to 1769472 tid 3 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:175: done (3 left) 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:180: copy_from oid 29 from oid 6 current snap is 14 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:174: finishing write tid 3 to smithi10536496-15 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:174: finishing write tid 4 to smithi10536496-15 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:174: finishing write tid 5 to smithi10536496-15 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 17 (ObjNum 87 snap 14 seq_num 87) dirty exists 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:174: left oid 15 (ObjNum 87 snap 14 seq_num 87) 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:178: finishing write tid 1 to smithi10536496-44 2024-09-15T03:41:29.040 INFO:tasks.rados.rados.0.smithi105.stdout:178: finishing write tid 2 to smithi10536496-44 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:178: finishing write tid 3 to smithi10536496-44 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:178: finishing write tid 4 to smithi10536496-44 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:174: done (3 left) 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:181: rollback oid 7 current snap is 14 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 7 to 13 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:182: rollback oid 16 current snap is 14 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 16 to 11 2024-09-15T03:41:29.041 INFO:tasks.rados.rados.0.smithi105.stdout:183: snap_remove snap 12 2024-09-15T03:41:29.050 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:29.048+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:29.057 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=14, to_read={3:0e027997:::smithi10536496-12:head=0,4096,0}, subchunks={3:0e027997:::smithi10536496-12:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.018463306000001, 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0095256449999999993, 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:41:29.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037696+0000", 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1408e-05 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046920+0000", 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092243440000000006 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046927+0000", 2024-09-15T03:41:29.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8090000000000004e-06 2024-09-15T03:41:29.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046974+0000", 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6869000000000001e-05 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047209+0000", 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235687 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:49 3.5s0 3:ac8f3d24:::smithi10536496-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.017694980999998, 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091978560000000008, 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 49 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038452+0000", 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038454+0000", 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7099999999999999e-06 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038626+0000", 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017214699999999999 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038636+0000", 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0199e-05 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038641+0000", 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.685e-06 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038679+0000", 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7731999999999997e-05 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038769+0000", 2024-09-15T03:41:29.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0315999999999994e-05 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.039430+0000", 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00066085199999999997 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040743+0000", 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001313346 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047587+0000", 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068434860000000002 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047650+0000", 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3382999999999999e-05 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.068 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:50 3.5s0 3:ac8f3d24:::smithi10536496-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.017497933000001, 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091805140000000007, 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 50 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.069 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038649+0000", 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038650+0000", 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6400000000000003e-07 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038652+0000", 2024-09-15T03:41:29.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.182e-06 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038656+0000", 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5090000000000004e-06 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038659+0000", 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4030000000000001e-06 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.039949+0000", 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001289884 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040025+0000", 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6005000000000002e-05 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00037365599999999999 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040954+0000", 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055585300000000003 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047765+0000", 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068113210000000004 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047830+0000", 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4237000000000002e-05 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:51 3.5s0 3:ac8f3d24:::smithi10536496-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.017472905000002, 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01164283, 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:29.075 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 51 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2599999999999998e-07 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038676+0000", 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.508e-06 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038680+0000", 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.653e-06 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038685+0000", 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9269999999999996e-06 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040567+0000", 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00188288 2024-09-15T03:41:29.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040733+0000", 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016570300000000001 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048135+0000", 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0074020889999999997 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048217+0000", 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2152999999999998e-05 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050317+0000", 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002099591 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=16, reqid=client.4562.0:65, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:29.080 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.008610203, 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020348845000000001, 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5899999999999996e-07 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047553+0000", 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6099999999999998e-05 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066726+0000", 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019172577 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066731+0000", 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3340000000000001e-06 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066780+0000", 2024-09-15T03:41:29.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8708999999999997e-05 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066790+0000", 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9709999999999999e-06 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067856+0000", 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001066259 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067886+0000", 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9335999999999998e-05 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:69 3.5s0 3:af19ecb4:::smithi10536496-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.007564558999999, 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094136830000000008, 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.085 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 69 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048583+0000", 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048585+0000", 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4729999999999999e-06 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048592+0000", 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2360000000000001e-06 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048596+0000", 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.092e-06 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051062+0000", 2024-09-15T03:41:29.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024665440000000002 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051144+0000", 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1369e-05 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056556+0000", 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054122759999999997 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056628+0000", 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1699999999999995e-05 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057996+0000", 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001368534 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.090 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=18, reqid=client.4562.0:86, at_version=35'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.988724788999999, 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057601141000000002, 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067423+0000", 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2499999999999995e-07 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067437+0000", 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4445e-05 2024-09-15T03:41:29.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121493+0000", 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054055923999999998 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121501+0000", 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8949999999999998e-06 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121585+0000", 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3502000000000001e-05 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121613+0000", 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8503000000000001e-05 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.124988+0000", 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033745590000000001 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125023+0000", 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5787999999999997e-05 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:114 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.958188968999998, 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093237049999999998, 2024-09-15T03:41:29.095 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 114 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1899999999999998e-07 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097960+0000", 2024-09-15T03:41:29.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6130000000000001e-06 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097966+0000", 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3690000000000003e-06 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097970+0000", 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0069999999999999e-06 2024-09-15T03:41:29.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099933+0000", 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019621830000000002 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.100014+0000", 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0852000000000004e-05 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104698+0000", 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046846140000000001 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104783+0000", 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4979000000000001e-05 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107282+0000", 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024986689999999998 2024-09-15T03:41:29.100 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=19, to_read={3:0215fa29:::smithi10536496-22:head=0,4096,0}, subchunks={3:0215fa29:::smithi10536496-22:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.934727234, 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039885337, 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121442+0000", 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2169999999999999e-05 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160985+0000", 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039542246000000003 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160992+0000", 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2840000000000001e-06 2024-09-15T03:41:29.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4248e-05 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161305+0000", 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025900800000000002 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.104 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.9s4 35/32 ECSubWrite(tid=25, reqid=client.4562.0:161, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.892092409, 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075462681000000004, 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:29.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9299999999999998e-07 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164071+0000", 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5299e-05 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238920+0000", 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074849009999999994 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238927+0000", 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4039999999999998e-06 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238986+0000", 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9086000000000003e-05 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238993+0000", 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3719999999999999e-06 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239496+0000", 2024-09-15T03:41:29.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050214699999999999 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239517+0000", 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.177e-05 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:289 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.028089624, 2024-09-15T03:41:29.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016317213000000001, 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 289 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028059+0000", 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3969999999999999e-06 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028066+0000", 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.627e-06 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028082+0000", 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5406000000000001e-05 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028087+0000", 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4809999999999997e-06 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028181+0000", 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3881999999999993e-05 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028280+0000", 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9135999999999994e-05 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028913+0000", 2024-09-15T03:41:29.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00063293699999999997 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032481+0000", 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003567677 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044305+0000", 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011823828 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044375+0000", 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9842e-05 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:290 3.5s0 3:a7103f89:::smithi10536496-39:head [rollback b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.028047229999999, 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016591980999999999, 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.115 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 290 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028104+0000", 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.501e-06 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028112+0000", 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2379999999999997e-06 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028116+0000", 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3290000000000004e-06 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029591+0000", 2024-09-15T03:41:29.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001475004 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029762+0000", 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000170557 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030258+0000", 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049562300000000005 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035019+0000", 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004761276 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044634+0000", 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096147730000000001 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044692+0000", 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8220000000000002e-05 2024-09-15T03:41:29.120 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:291 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.028026061999999, 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019516250999999998, 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.121 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 291 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028123+0000", 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1239999999999999e-06 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028128+0000", 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5449999999999997e-06 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028132+0000", 2024-09-15T03:41:29.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0090000000000001e-06 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030541+0000", 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024089659999999998 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030601+0000", 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0384000000000003e-05 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.125 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030921+0000", 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031995500000000002 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035034+0000", 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0041124630000000002 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047571+0000", 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012537510999999999 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047637+0000", 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5925000000000003e-05 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=22, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.019154922999999, 2024-09-15T03:41:29.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011072657, 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1899999999999999e-07 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036994+0000", 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047471+0000", 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010477119 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047475+0000", 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3989999999999997e-06 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047509+0000", 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3426000000000002e-05 2024-09-15T03:41:29.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.048065+0000", 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055612399999999996 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=23, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.003384067999999, 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92706636600000003, 2024-09-15T03:41:29.131 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9599999999999999e-07 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052767+0000", 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8199999999999998e-06 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979592+0000", 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92682499399999996 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979604+0000", 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1387e-05 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979726+0000", 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122014 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979829+0000", 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103655 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:312 3.5s0 3:a1c61493:::smithi10536496-38:head [append 0~622592] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.057916+0000", 2024-09-15T03:41:29.135 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.998231452000001, 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0083523039999999996, 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 312 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.057916+0000", 2024-09-15T03:41:29.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.057916+0000", 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.057917+0000", 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.787e-06 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058334+0000", 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041702299999999998 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058348+0000", 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4079e-05 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058355+0000", 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5200000000000003e-06 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058401+0000", 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5565999999999997e-05 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058545+0000", 2024-09-15T03:41:29.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014402999999999999 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059495+0000", 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095007399999999997 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.066126+0000", 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066316860000000004 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.066198+0000", 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1996999999999995e-05 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.066268+0000", 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9542000000000006e-05 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.141 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:313 3.5s0 3:a1c61493:::smithi10536496-38:head [append 0~147456] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.997189332, 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0098668620000000005, 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 313 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.142 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058959+0000", 2024-09-15T03:41:29.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.108e-06 2024-09-15T03:41:29.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059032+0000", 2024-09-15T03:41:29.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3231000000000004e-05 2024-09-15T03:41:29.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059047+0000", 2024-09-15T03:41:29.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4829e-05 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059053+0000", 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5910000000000001e-06 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.060287+0000", 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012338760000000001 2024-09-15T03:41:29.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.060443+0000", 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015520099999999999 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.061055+0000", 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00061251899999999995 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068673+0000", 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076178460000000002 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068754+0000", 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1104000000000001e-05 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068825+0000", 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0556999999999995e-05 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:314 3.5s0 3:a1c61493:::smithi10536496-38:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:29.149 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.99708624, 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015366338, 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 314 2024-09-15T03:41:29.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.151 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:41:29.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-07 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059066+0000", 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8790000000000002e-06 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059075+0000", 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6009999999999995e-06 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059079+0000", 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999997e-06 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.063568+0000", 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044885849999999998 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.063652+0000", 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4670000000000001e-05 2024-09-15T03:41:29.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.064159+0000", 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050645400000000004 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068840+0000", 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046815729999999996 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074353+0000", 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055131449999999997 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074427+0000", 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3767e-05 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:315 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:41:29.156 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.997063566, 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018255761999999998, 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 315 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:41:29.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059086+0000", 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1390000000000002e-06 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3779999999999998e-06 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059098+0000", 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.065037+0000", 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005938484 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.065126+0000", 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9343000000000002e-05 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074804+0000", 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096778360000000004 2024-09-15T03:41:29.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074926+0000", 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122092 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.077339+0000", 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024133190000000001 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:325 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.973137571000001, 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.078787472999999997, 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 325 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.162 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083010+0000", 2024-09-15T03:41:29.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.46e-07 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083012+0000", 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083020+0000", 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5769999999999996e-06 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083024+0000", 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.143e-06 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085552+0000", 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025276999999999999 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085630+0000", 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7831999999999996e-05 2024-09-15T03:41:29.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086020+0000", 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039002200000000001 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.090715+0000", 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046956599999999999 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:29.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.161716+0000", 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071000895999999994 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.161797+0000", 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0729000000000006e-05 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:29.167 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:29.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:41:29.169 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:29.169+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:29.170 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:29.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:29.277 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:29.277+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:29.284 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:29.284 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:29.284 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:29.284 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:29.295 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:29.371 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:29.370+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:29.388 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:29.427 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:29.427 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:29.439 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:29.439 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:29.448 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:29.448 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:29.449 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:29.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:29.469 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:29.469 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:29.471 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:29.471+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:29.489 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:29.489 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:29.510 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:29.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:29.530 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:29.531 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:29.550 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:29.550 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:29.551 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:29.551 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:29.558 INFO:teuthology.orchestra.run.smithi105.stdout:fsck success 2024-09-15T03:41:29.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:29.565 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-15T03:41:29.566 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-15T03:41:29.566 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:41:29.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:41:29.568 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:41:29.569 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:29.570 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:29.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:29.573 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:29.570+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:29.591 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:29.591 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:29.612 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:29.632 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:29.633 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:29.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:29.653 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:no valid command found; 10 closest matches: 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:0 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:1 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:2 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:abort 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:assert 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:config diff 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:config diff get 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:config get 2024-09-15T03:41:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:config help [] 2024-09-15T03:41:29.664 INFO:teuthology.orchestra.run.smithi105.stderr:config set ... 2024-09-15T03:41:29.664 INFO:teuthology.orchestra.run.smithi105.stderr:admin_socket: invalid command 2024-09-15T03:41:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T03:41:29.665 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-15T03:41:29.673 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:29.673+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:29.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:29.674 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:29.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:29.674+0000 7f7bf3fff640 0 -- 172.21.15.5:0/1652769918 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f7bfc195380 msgr2=0x7f7bfc197760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:29.675 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:29.675 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:29.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:29.775 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:29.774+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:29.826 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:29.826 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:29.834 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:29.875 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:29.874+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:29.910 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:29.911 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:29.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:29.947 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:29.948 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:29.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:29.976 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:29.975+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:30.002 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:41:30.004 INFO:tasks.rados.rados.0.smithi105.stdout:183: done (5 left) 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:184: delete oid 25 current snap is 14 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:181: finishing rollback tid 1 to smithi10536496-7 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:181: finishing rollback tid 2 to smithi10536496-7 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 18 (ObjNum 82 snap 9 seq_num 82) dirty exists 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:182: finishing rollback tid 0 to smithi10536496-16 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:180: finishing copy_from racing read to smithi10536496-29 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:182: finishing rollback tid 1 to smithi10536496-16 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:182: finishing rollback tid 2 to smithi10536496-16 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 34 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:179: finishing write tid 1 to smithi10536496-36 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:179: finishing write tid 2 to smithi10536496-36 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:180: finishing copy_from to smithi10536496-29 2024-09-15T03:41:30.005 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:179: finishing write tid 3 to smithi10536496-36 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:179: finishing write tid 4 to smithi10536496-36 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:179: finishing write tid 5 to smithi10536496-36 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 28 (ObjNum 90 snap 14 seq_num 90) dirty exists 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:179: left oid 36 (ObjNum 90 snap 14 seq_num 90) 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:178: finishing write tid 5 to smithi10536496-44 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 38 (ObjNum 89 snap 14 seq_num 89) dirty exists 2024-09-15T03:41:30.006 INFO:tasks.rados.rados.0.smithi105.stdout:178: left oid 44 (ObjNum 89 snap 14 seq_num 89) 2024-09-15T03:41:30.015 INFO:tasks.rados.rados.0.smithi105.stdout:178: done (5 left) 2024-09-15T03:41:30.015 INFO:tasks.rados.rados.0.smithi105.stdout:179: done (4 left) 2024-09-15T03:41:30.015 INFO:tasks.rados.rados.0.smithi105.stdout:180: done (3 left) 2024-09-15T03:41:30.015 INFO:tasks.rados.rados.0.smithi105.stdout:181: done (2 left) 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:182: done (1 left) 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:184: done (0 left) 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:185: rollback oid 13 current snap is 14 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 13 to 13 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:186: read oid 12 snap 11 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:186: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T03:41:30.016 INFO:tasks.rados.rados.0.smithi105.stdout:187: delete oid 50 current snap is 14 2024-09-15T03:41:30.019 INFO:tasks.rados.rados.0.smithi105.stdout:186: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-15T03:41:30.028 INFO:tasks.rados.rados.0.smithi105.stdout:186: done (2 left) 2024-09-15T03:41:30.028 INFO:tasks.rados.rados.0.smithi105.stdout:187: done (1 left) 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:188: read oid 43 snap -1 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:188: expect deleted 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:189: delete oid 30 current snap is 14 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:185: finishing rollback tid 0 to smithi10536496-13 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:185: finishing rollback tid 1 to smithi10536496-13 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:185: finishing rollback tid 2 to smithi10536496-13 2024-09-15T03:41:30.029 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 32 (ObjNum 69 snap 5 seq_num 69) dirty exists 2024-09-15T03:41:30.034 INFO:tasks.rados.rados.0.smithi105.stdout:185: done (2 left) 2024-09-15T03:41:30.034 INFO:tasks.rados.rados.0.smithi105.stdout:188: done (1 left) 2024-09-15T03:41:30.034 INFO:tasks.rados.rados.0.smithi105.stdout:189: done (0 left) 2024-09-15T03:41:30.034 INFO:tasks.rados.rados.0.smithi105.stdout:190: copy_from oid 20 from oid 27 current snap is 14 2024-09-15T03:41:30.035 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 12 current snap is 14 2024-09-15T03:41:30.035 INFO:tasks.rados.rados.0.smithi105.stdout:191: seq_num 94 ranges {196608=425984,622592=786432,1409024=376832} 2024-09-15T03:41:30.037 INFO:tasks.rados.rados.0.smithi105.stdout:191: writing smithi10536496-12 from 196608 to 622592 tid 1 2024-09-15T03:41:30.042 INFO:tasks.rados.rados.0.smithi105.stdout:191: writing smithi10536496-12 from 622592 to 1409024 tid 2 2024-09-15T03:41:30.045 INFO:tasks.rados.rados.0.smithi105.stdout:191: writing smithi10536496-12 from 1409024 to 1785856 tid 3 2024-09-15T03:41:30.045 INFO:tasks.rados.rados.0.smithi105.stdout:192: rmattr oid 47 current snap is 14 2024-09-15T03:41:30.045 INFO:tasks.rados.rados.0.smithi105.stdout:192: done (2 left) 2024-09-15T03:41:30.045 INFO:tasks.rados.rados.0.smithi105.stdout:193: snap_remove snap 13 2024-09-15T03:41:30.077 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:30.075+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:30.112 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.112 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.120 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:30.153 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:30.151+0000 7fd248430440 -1 Falling back to public interface 2024-09-15T03:41:30.178 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:30.176+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:30.222 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.222 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.231 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:30.278 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:30.278+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:30.333 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.333 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:30.385 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:30.385+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:30.444 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.444 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:30.479 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:30.478+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:30.580 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:30.579+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:30.595 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.596 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.604 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:30.681 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:30.681+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:30.705 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.705 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:30.783 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:30.782+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:30.832 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.832 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:30.886 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:30.885+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:30.945 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:30.945 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:30.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:30.985 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:30.984+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:193: done (2 left) 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:194: delete oid 2 current snap is 14 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:191: finishing write tid 1 to smithi10536496-12 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:190: finishing copy_from racing read to smithi10536496-20 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:190: finishing copy_from to smithi10536496-20 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:191: finishing write tid 2 to smithi10536496-12 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:191: finishing write tid 3 to smithi10536496-12 2024-09-15T03:41:31.008 INFO:tasks.rados.rados.0.smithi105.stdout:191: finishing write tid 4 to smithi10536496-12 2024-09-15T03:41:31.009 INFO:tasks.rados.rados.0.smithi105.stdout:191: finishing write tid 5 to smithi10536496-12 2024-09-15T03:41:31.009 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 40 (ObjNum 94 snap 14 seq_num 94) dirty exists 2024-09-15T03:41:31.009 INFO:tasks.rados.rados.0.smithi105.stdout:191: left oid 12 (ObjNum 94 snap 14 seq_num 94) 2024-09-15T03:41:31.013 INFO:tasks.rados.rados.0.smithi105.stdout:190: done (2 left) 2024-09-15T03:41:31.013 INFO:tasks.rados.rados.0.smithi105.stdout:191: done (1 left) 2024-09-15T03:41:31.013 INFO:tasks.rados.rados.0.smithi105.stdout:194: done (0 left) 2024-09-15T03:41:31.013 INFO:tasks.rados.rados.0.smithi105.stdout:195: delete oid 11 current snap is 14 2024-09-15T03:41:31.029 INFO:tasks.rados.rados.0.smithi105.stdout:195: done (0 left) 2024-09-15T03:41:31.029 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 25 current snap is 14 2024-09-15T03:41:31.029 INFO:tasks.rados.rados.0.smithi105.stdout:196: seq_num 97 ranges {0=475136,475136=442368,917504=442368,1359872=278528} 2024-09-15T03:41:31.031 INFO:tasks.rados.rados.0.smithi105.stdout:196: writing smithi10536496-25 from 0 to 475136 tid 1 2024-09-15T03:41:31.034 INFO:tasks.rados.rados.0.smithi105.stdout:196: writing smithi10536496-25 from 475136 to 917504 tid 2 2024-09-15T03:41:31.036 INFO:tasks.rados.rados.0.smithi105.stdout:196: writing smithi10536496-25 from 917504 to 1359872 tid 3 2024-09-15T03:41:31.038 INFO:tasks.rados.rados.0.smithi105.stdout:196: writing smithi10536496-25 from 1359872 to 1638400 tid 4 2024-09-15T03:41:31.038 INFO:tasks.rados.rados.0.smithi105.stdout:197: snap_remove snap 11 2024-09-15T03:41:31.087 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:31.085+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:31.105 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.105 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:31.187 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:31.187+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:31.215 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.215 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:31.288 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:31.287+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:31.324 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.325 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:31.390 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:31.388+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:31.434 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.434 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:31.490 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:31.490+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:31.544 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.544 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:31.592 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:31.591+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:31.654 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.654 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:31.707 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:31.706+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:31.765 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.765 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:31.815 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:31.814+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:31.894 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:31.894+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:31.915 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:31.916 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:31.916 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:31.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:31.995 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:31.994+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:197: done (1 left) 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:198: snap_create 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing write tid 1 to smithi10536496-25 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing write tid 2 to smithi10536496-25 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing write tid 3 to smithi10536496-25 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing write tid 4 to smithi10536496-25 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing write tid 5 to smithi10536496-25 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing write tid 6 to smithi10536496-25 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 32 (ObjNum 97 snap 14 seq_num 97) dirty exists 2024-09-15T03:41:32.013 INFO:tasks.rados.rados.0.smithi105.stdout:196: left oid 25 (ObjNum 97 snap 14 seq_num 97) 2024-09-15T03:41:32.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:41:32.035 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:32.035 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:32.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:32.066 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:32.066 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:32.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:32.087 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:32.097 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:32.094+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:32.107 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:32.108 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:32.108 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:32.127 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:32.128 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:32.148 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:32.149 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:32.149 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:32.149 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:32.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:32.168 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:32.168 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:32.188 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:32.188 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:32.199 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:32.198+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:32.208 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:32.208 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:32.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:32.228 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:32.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:32.250 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:32.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:32.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:32.259 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:32.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:32.267 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:32.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:32.269 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:32.288 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:32.289 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:32.308 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:32.309 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:32.311 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:32.310+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:32.331 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:32.331 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:32.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:32.335+0000 7fa04259c640 0 -- 172.21.15.5:0/4067682472 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7fa0441930e0 msgr2=0x7fa0441954c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:32.361 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:32.361 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:32.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:32.366 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:32.366 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:32.366 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:32.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:32.405 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:32.405+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:32.502 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:32.501+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:32.515 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:32.515 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:32.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:32.515 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:32.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4185.0:66 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'44, mlcod=24'40)", 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 48.500866537, 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077416819999999997, 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012540+0000", 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012541+0000", 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2e-07 2024-09-15T03:41:32.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012592+0000", 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1273999999999997e-05 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012612+0000", 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0389000000000001e-05 2024-09-15T03:41:32.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012615+0000", 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1250000000000001e-06 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012636+0000", 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1070999999999999e-05 2024-09-15T03:41:32.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.012647+0000", 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.08e-05 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020245+0000", 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075975579999999999 2024-09-15T03:41:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:40:44.020282+0000", 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7045000000000002e-05 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:5 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.496773624999999, 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010334054000000001, 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016634+0000", 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7000000000000004e-07 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016637+0000", 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6280000000000001e-06 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8289999999999997e-06 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016649+0000", 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017615+0000", 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096610800000000003 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017698+0000", 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3428000000000002e-05 2024-09-15T03:41:32.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018129+0000", 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043087300000000002 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022895+0000", 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047655759999999997 2024-09-15T03:41:32.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026926+0000", 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040308160000000004 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026967+0000", 2024-09-15T03:41:32.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1554999999999998e-05 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:6 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.496741723, 2024-09-15T03:41:32.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012850047, 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6100000000000001e-07 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016667+0000", 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9769999999999999e-06 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016674+0000", 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016678+0000", 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000003e-06 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018325+0000", 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001646385 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6032999999999995e-05 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027234+0000", 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088331369999999996 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027316+0000", 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1862999999999999e-05 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029515+0000", 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002199281 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s5 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.474148453000002, 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079303010000000007, 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1000000000000001e-07 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039586+0000", 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032722799999999999 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039615+0000", 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9383e-05 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039621+0000", 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1029999999999998e-06 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039673+0000", 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2261e-05 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039682+0000", 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7649999999999999e-06 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047166+0000", 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074841710000000004 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047189+0000", 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.228e-05 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:58 3.es0 3:7dd49206:::smithi10536496-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.473117934000001, 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083733009999999997, 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1990000000000001e-06 2024-09-15T03:41:32.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040639+0000", 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034862499999999998 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040646+0000", 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6290000000000001e-06 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040650+0000", 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.552e-06 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040691+0000", 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.07e-05 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040781+0000", 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0859000000000005e-05 2024-09-15T03:41:32.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041560+0000", 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00077845500000000005 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044707+0000", 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031467969999999998 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048626+0000", 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039196150000000004 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048662+0000", 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.587e-05 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:59 3.es0 3:7dd49206:::smithi10536496-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.472750421000001, 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084161029999999994, 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 59 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040657+0000", 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4899999999999995e-07 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040659+0000", 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3319999999999999e-06 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040665+0000", 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6690000000000003e-06 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040668+0000", 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4630000000000001e-06 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041990+0000", 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013215220000000001 2024-09-15T03:41:32.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042074+0000", 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4060000000000005e-05 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042441+0000", 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000367432 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047193+0000", 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047516440000000002 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049024+0000", 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001831378 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049072+0000", 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8053999999999998e-05 2024-09-15T03:41:32.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:60 3.es0 3:7dd49206:::smithi10536496-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.472734631000002, 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010867740000000001, 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 60 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040673+0000", 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040675+0000", 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2479999999999999e-06 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040681+0000", 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0170000000000002e-06 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040689+0000", 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6330000000000002e-06 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042594+0000", 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019041489999999999 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042676+0000", 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2633999999999995e-05 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049349+0000", 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066727380000000001 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049419+0000", 2024-09-15T03:41:32.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9868000000000003e-05 2024-09-15T03:41:32.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.051540+0000", 2024-09-15T03:41:32.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002121029 2024-09-15T03:41:32.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:83 3.es0 3:7f98869d:::smithi10536496-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.450763712000001, 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080589019999999997, 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 83 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:32.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062645+0000", 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9199999999999998e-06 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062651+0000", 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062654+0000", 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6729999999999998e-06 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.063750+0000", 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001095525 2024-09-15T03:41:32.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.063801+0000", 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1573000000000002e-05 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064048+0000", 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024709100000000001 2024-09-15T03:41:32.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064653+0000", 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060490800000000003 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070635+0000", 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059821070000000004 2024-09-15T03:41:32.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070702+0000", 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6548e-05 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:84 3.es0 3:7f98869d:::smithi10536496-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:32.566 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.450729498000001, 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011714999, 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 84 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:32.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062678+0000", 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062679+0000", 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6920000000000001e-06 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062685+0000", 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0979999999999999e-06 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062688+0000", 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3860000000000001e-06 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064229+0000", 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015411999999999999 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064308+0000", 2024-09-15T03:41:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8621000000000005e-05 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070977+0000", 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006669538 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.071049+0000", 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1464000000000006e-05 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.074392+0000", 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003343586 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.3s4 35/32 ECSubWrite(tid=126, reqid=client.4562.0:134, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.380955907000001, 2024-09-15T03:41:32.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078231129999999996, 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:32.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132451+0000", 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.44e-07 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132468+0000", 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6424999999999999e-05 2024-09-15T03:41:32.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132496+0000", 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7798e-05 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132500+0000", 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2930000000000002e-06 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132531+0000", 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1028999999999997e-05 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.132537+0000", 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.699e-06 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.140252+0000", 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007715431 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.140274+0000", 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1793999999999998e-05 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:152 3.es0 3:71615b7a:::smithi10536496-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.357794458000001, 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.031528529, 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 152 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:32.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155613+0000", 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155616+0000", 2024-09-15T03:41:32.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.88e-06 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155623+0000", 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6950000000000003e-06 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155627+0000", 2024-09-15T03:41:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999997e-06 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156746+0000", 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001119229 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156828+0000", 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2294999999999997e-05 2024-09-15T03:41:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157113+0000", 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028439999999999997 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.159322+0000", 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022090719999999999 2024-09-15T03:41:32.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187068+0000", 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027746694999999998 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187141+0000", 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-09-15T03:41:32.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:153 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.357773108, 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.032862542000000002, 2024-09-15T03:41:32.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 153 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:32.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155636+0000", 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7260000000000001e-06 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155642+0000", 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1449999999999996e-06 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155645+0000", 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4470000000000002e-06 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157320+0000", 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016747769999999999 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157366+0000", 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5982999999999997e-05 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187194+0000", 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.029827955 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187240+0000", 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5494000000000002e-05 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188496+0000", 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012565250000000001 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.591 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:166 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.327887617000002, 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.10104360799999999, 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.592 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 166 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:32.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185520+0000", 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4099999999999999e-07 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185522+0000", 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.588e-06 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185526+0000", 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5130000000000001e-06 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185528+0000", 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3099999999999999e-06 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186513+0000", 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098462300000000001 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8233000000000003e-05 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186810+0000", 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023878899999999999 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187938+0000", 2024-09-15T03:41:32.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001128698 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286460+0000", 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098521759 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286563+0000", 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102654 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:167 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.327868344999999, 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.101989948, 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 167 2024-09-15T03:41:32.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185539+0000", 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8599999999999999e-07 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185540+0000", 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4020000000000001e-06 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185544+0000", 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6619999999999998e-06 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185547+0000", 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6330000000000002e-06 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188004+0000", 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024565619999999998 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188048+0000", 2024-09-15T03:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3925999999999999e-05 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286831+0000", 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098782707999999997 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286925+0000", 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4609000000000002e-05 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.287528+0000", 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060316000000000005 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s2 45/44 ECSubWrite(tid=22, reqid=client.4562.0:286, at_version=45'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.031284+0000", 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.482122741000001, 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084639229999999999, 2024-09-15T03:41:32.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031284+0000", 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031284+0000", 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031288+0000", 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5300000000000001e-06 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031727+0000", 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043953200000000002 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031749+0000", 2024-09-15T03:41:32.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1617e-05 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031758+0000", 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7489999999999993e-06 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031808+0000", 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9103000000000001e-05 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031818+0000", 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0273e-05 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.039718+0000", 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079004350000000008 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.039748+0000", 2024-09-15T03:41:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9683999999999999e-05 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.10s1 45/44 ECSubWrite(tid=39, reqid=client.4562.0:287, at_version=45'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.456276994, 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30713189699999999, 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057131+0000", 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3799999999999996e-07 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057345+0000", 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021436799999999999 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362921+0000", 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30557607399999998 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362927+0000", 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8549999999999998e-06 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362967+0000", 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9654999999999999e-05 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362974+0000", 2024-09-15T03:41:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.364234+0000", 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012599200000000001 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.364262+0000", 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7631e-05 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.612 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.es0 54/32 ECSubWriteReply(tid=55, last_complete=54'26, committed=1, applied=1))", 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.455736665, 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.195800472, 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057673+0000", 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.751e-06 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253340+0000", 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.19566710400000001 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253347+0000", 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1250000000000004e-06 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253426+0000", 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8955999999999994e-05 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253471+0000", 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4121e-05 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s2 62/44 ECSubWrite(tid=65, reqid=client.4562.0:441, at_version=62'28, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047425+0000", 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.4659815649999999, 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077610029999999998, 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.617 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047425+0000", 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047425+0000", 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047426+0000", 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0999999999999998e-07 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047444+0000", 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8394e-05 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047483+0000", 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8782000000000002e-05 2024-09-15T03:41:32.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047490+0000", 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0650000000000001e-06 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047562+0000", 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1746000000000001e-05 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047574+0000", 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2089000000000001e-05 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.055166+0000", 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075924290000000004 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.055186+0000", 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9788000000000001e-05 2024-09-15T03:41:32.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s1 62/44 ECSubWrite(tid=41, reqid=client.4562.0:443, at_version=62'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.4642544700000002, 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099932440000000001, 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.29e-07 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049191+0000", 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8832999999999999e-05 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049237+0000", 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5318000000000002e-05 2024-09-15T03:41:32.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049243+0000", 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.922e-06 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049303+0000", 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0004999999999999e-05 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049314+0000", 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0779e-05 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059119+0000", 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098048660000000006 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059146+0000", 2024-09-15T03:41:32.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6392000000000001e-05 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s2 62/44 ECSubWrite(tid=66, reqid=client.4562.0:441, at_version=0'0, trim_to=0'0, roll_forward_to=62'28))", 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.064007+0000", 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.449399723, 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007908155, 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.627 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064007+0000", 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064007+0000", 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064008+0000", 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6700000000000003e-07 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064032+0000", 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4142999999999999e-05 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064065+0000", 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3136000000000003e-05 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064071+0000", 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2899999999999999e-06 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064119+0000", 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8092e-05 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064130+0000", 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0665999999999999e-05 2024-09-15T03:41:32.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.071847+0000", 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077167080000000001 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.071915+0000", 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8453000000000003e-05 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.632 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.632 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:32.633 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:32.564+0000 7fd248430440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:41:32.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:32.634 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:32.632+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:32.734 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:32.733+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:32.772 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:32.772 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:32.772 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:32.773 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:32.780 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:32.835 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:32.834+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:32.877 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:32.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:32.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:32.935 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:32.935+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=0, reqid=client.4562.0:13, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.962126706999999, 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011510825000000001, 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019689+0000", 2024-09-15T03:41:32.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5799999999999998e-07 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019738+0000", 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8652000000000001e-05 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019844+0000", 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106623 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019852+0000", 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019952+0000", 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100111 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019961+0000", 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:41:32.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031174+0000", 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011212744 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031199+0000", 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5278000000000001e-05 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.962065109000001, 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011454499999999999, 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019805+0000", 2024-09-15T03:41:32.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5062000000000002e-05 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020203+0000", 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039759399999999997 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020209+0000", 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6189999999999999e-06 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020263+0000", 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4410000000000003e-05 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020283+0000", 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0176000000000001e-05 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031185+0000", 2024-09-15T03:41:32.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010901569 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031204+0000", 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9663000000000001e-05 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=2, reqid=client.4562.0:14, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.961260631999998, 2024-09-15T03:41:32.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010659454, 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:32.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020555+0000", 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-07 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020569+0000", 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3984e-05 2024-09-15T03:41:32.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020610+0000", 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1411e-05 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020616+0000", 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1809999999999998e-06 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022402+0000", 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001785972 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022411+0000", 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6920000000000008e-06 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:32.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031203+0000", 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087921389999999992 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031214+0000", 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0902e-05 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:32.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.960217249999999, 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012719667, 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:32.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021599+0000", 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021655+0000", 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6644999999999997e-05 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021703+0000", 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7549999999999997e-05 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021711+0000", 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1519999999999992e-06 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022571+0000", 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086020699999999999 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022577+0000", 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3339999999999998e-06 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034283+0000", 2024-09-15T03:41:33.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011705224 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034317+0000", 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.485e-05 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.004 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.95949482, 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012060876, 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:33.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022321+0000", 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:41:33.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023195+0000", 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087440700000000001 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023229+0000", 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3905000000000001e-05 2024-09-15T03:41:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023237+0000", 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.43e-06 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023278+0000", 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1137999999999998e-05 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023286+0000", 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5379999999999997e-06 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034356+0000", 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01107022 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034381+0000", 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5395999999999999e-05 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:33.010 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.958333823, 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010863246999999999, 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023482+0000", 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023498+0000", 2024-09-15T03:41:33.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6399e-05 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023536+0000", 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7944000000000001e-05 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023543+0000", 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023580+0000", 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7379999999999998e-05 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023588+0000", 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3950000000000001e-06 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034332+0000", 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010743068 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034344+0000", 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2918000000000001e-05 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.955482041, 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018217534000000001, 2024-09-15T03:41:33.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026334+0000", 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8500000000000001e-07 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026371+0000", 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6835999999999998e-05 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:41:33.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010326484 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036758+0000", 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0989999999999997e-05 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036813+0000", 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050000000000003e-05 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036822+0000", 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3149999999999998e-06 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044526+0000", 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077039439999999999 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:41:33.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.423e-05 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.944650513999999, 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087714329999999993, 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:33.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037181+0000", 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5954999999999999e-05 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122876+0000", 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085695108000000006 2024-09-15T03:41:33.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122883+0000", 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4069999999999998e-06 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122927+0000", 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552999999999998e-05 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122937+0000", 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.006e-05 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124855+0000", 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001917687 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124879+0000", 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4002000000000002e-05 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:305 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:41:33.025 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.963950343, 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057908950000001, 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 305 2024-09-15T03:41:33.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017866+0000", 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.468e-06 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017871+0000", 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8949999999999997e-06 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017882+0000", 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0781000000000001e-05 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017888+0000", 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7030000000000003e-06 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018332+0000", 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000444633 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018335+0000", 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7690000000000001e-06 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023618+0000", 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.005282727 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023656+0000", 2024-09-15T03:41:33.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7919000000000001e-05 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:306 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.963918332999999, 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057866579999999, 2024-09-15T03:41:33.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 306 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:33.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:41:33.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017900+0000", 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.756e-06 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017907+0000", 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017912+0000", 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4009999999999996e-06 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018381+0000", 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000468696 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018383+0000", 2024-09-15T03:41:33.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.906e-06 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023676+0000", 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0052937150000001 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023683+0000", 2024-09-15T03:41:33.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0550000000000002e-06 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:307 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.963898712999999, 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.00579789, 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 307 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:41:33.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017917+0000", 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:41:33.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017920+0000", 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3500000000000001e-06 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017926+0000", 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8529999999999997e-06 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017929+0000", 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3100000000000001e-06 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018419+0000", 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048992199999999995 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018420+0000", 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.313e-06 2024-09-15T03:41:33.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023709+0000", 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.005288537 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023714+0000", 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2310000000000004e-06 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s6 50/44 ECSubWrite(tid=31, reqid=client.4562.0:314, at_version=0'0, trim_to=0'0, roll_forward_to=50'20))", 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.907241295, 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011396837, 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074585+0000", 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0251999999999999e-05 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083831+0000", 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092458939999999993 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083835+0000", 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5430000000000004e-06 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083884+0000", 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9052999999999999e-05 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083894+0000", 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0246e-05 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.085938+0000", 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020431540000000002 2024-09-15T03:41:33.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.085971+0000", 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3030000000000001e-05 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s6 50/44 ECSubWrite(tid=35, reqid=client.4562.0:325, at_version=50'24, trim_to=0'0, roll_forward_to=50'20))", 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.895655239, 2024-09-15T03:41:33.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075270610000000002, 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086179+0000", 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8609000000000001e-05 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160878+0000", 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074699163999999998 2024-09-15T03:41:33.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160884+0000", 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6760000000000004e-06 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160935+0000", 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.117e-05 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160948+0000", 2024-09-15T03:41:33.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3355e-05 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.161407+0000", 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045902099999999999 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.161430+0000", 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2968000000000001e-05 2024-09-15T03:41:33.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDScrubReserve(3.7s1 REQUEST e60)", 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5127911760000003, 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120973636, 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3400000000000001e-07 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469026+0000", 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3530000000000002e-06 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589898+0000", 2024-09-15T03:41:33.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120871944 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589906+0000", 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5660000000000001e-06 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589955+0000", 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9027999999999999e-05 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589973+0000", 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8318e-05 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589998+0000", 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4292999999999999e-05 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=62, reqid=client.4562.0:438, at_version=62'25, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9403276739999997, 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017875489000000001, 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041488+0000", 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-09-15T03:41:33.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042539+0000", 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001050823 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042580+0000", 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0908000000000002e-05 2024-09-15T03:41:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042587+0000", 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7929999999999998e-06 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042636+0000", 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8996999999999999e-05 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042645+0000", 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0899999999999994e-06 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059315+0000", 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016669970999999999 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059363+0000", 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7503999999999998e-05 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=63, reqid=client.4562.0:439, at_version=62'26, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:33.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9361162260000002, 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013689702, 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045881+0000", 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018117900000000001 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045908+0000", 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7741000000000001e-05 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045913+0000", 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0370000000000002e-06 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047115+0000", 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012019629999999999 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047126+0000", 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0210000000000001e-05 2024-09-15T03:41:33.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059370+0000", 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01224459 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059389+0000", 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8328000000000001e-05 2024-09-15T03:41:33.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=64, reqid=client.4562.0:440, at_version=62'27, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9355135250000002, 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016955748, 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.069 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046339+0000", 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7045999999999997e-05 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046361+0000", 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2115000000000001e-05 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046366+0000", 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2290000000000002e-06 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047844+0000", 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001477864 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047853+0000", 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6559999999999998e-06 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063237+0000", 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015383681 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063257+0000", 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0645999999999999e-05 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 62/32 ECSubWrite(tid=96, reqid=client.4562.0:446, at_version=0'0, trim_to=0'0, roll_forward_to=62'18))", 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.934311584, 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011862770999999999, 2024-09-15T03:41:33.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047504+0000", 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5600000000000004e-07 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047516+0000", 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1987999999999999e-05 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047611+0000", 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4741000000000007e-05 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047622+0000", 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1270999999999999e-05 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047672+0000", 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.99e-05 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047718+0000", 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5705999999999999e-05 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059338+0000", 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011619643000000001 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059366+0000", 2024-09-15T03:41:33.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8666e-05 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=65, reqid=client.4562.0:441, at_version=62'28, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9339348030000001, 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015392342999999999, 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.079 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047881+0000", 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8700000000000004e-07 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047900+0000", 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8895000000000001e-05 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047932+0000", 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1656999999999999e-05 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047939+0000", 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8329999999999996e-06 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051450+0000", 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035115120000000001 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051461+0000", 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1439e-05 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063261+0000", 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011799924999999999 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063273+0000", 2024-09-15T03:41:33.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1195e-05 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 62/44 ECSubWrite(tid=41, reqid=client.4562.0:443, at_version=62'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9325306790000001, 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014072049999999999, 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.084 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049285+0000", 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049336+0000", 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1023000000000003e-05 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049377+0000", 2024-09-15T03:41:33.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1591000000000002e-05 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049382+0000", 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9609999999999996e-06 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051693+0000", 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023109720000000001 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051702+0000", 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9430000000000006e-06 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063337+0000", 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011634698000000001 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063356+0000", 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9599e-05 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.089 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:33.091 INFO:tasks.rados.rados.0.smithi105.stdout:196: done (1 left) 2024-09-15T03:41:33.091 INFO:tasks.rados.rados.0.smithi105.stdout:198: done (0 left) 2024-09-15T03:41:33.091 INFO:tasks.rados.rados.0.smithi105.stdout:199: rmattr oid 30 current snap is 15 2024-09-15T03:41:33.091 INFO:tasks.rados.rados.0.smithi105.stdout:199: done (0 left) 2024-09-15T03:41:33.091 INFO:tasks.rados.rados.0.smithi105.stdout:200: snap_remove snap 14 2024-09-15T03:41:33.092 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:41:33.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:33.093 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:33.090+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:33.192 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:33.191+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:33.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:33.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:33.233 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:33.233 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:33.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:33.292 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:33.292+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:33.329 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:33.326+0000 7fd248430440 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:41:33.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:33.380 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:33.380 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:33.380 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:33.380 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:33.386 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:33.394 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:33.393+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:33.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:33.486 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.463886122000002, 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008773889, 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:33.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019951+0000", 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7080000000000002e-05 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020259+0000", 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030882000000000001 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020269+0000", 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4690000000000003e-06 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020353+0000", 2024-09-15T03:41:33.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4121000000000003e-05 2024-09-15T03:41:33.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020367+0000", 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.401e-05 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028603+0000", 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082364340000000008 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028667+0000", 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3665999999999997e-05 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.463038845, 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075521720000000002, 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.493 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020740+0000", 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020741+0000", 2024-09-15T03:41:33.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0399999999999995e-07 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020828+0000", 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7108999999999996e-05 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020883+0000", 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4818000000000003e-05 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020888+0000", 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.763e-06 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020921+0000", 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2780000000000001e-05 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020926+0000", 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8359999999999997e-06 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028240+0000", 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073138810000000004 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028293+0000", 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2281000000000003e-05 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=1, reqid=client.4562.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.462946994999999, 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075072999999999997, 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020832+0000", 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020833+0000", 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4400000000000001e-07 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020843+0000", 2024-09-15T03:41:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020899+0000", 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6416000000000003e-05 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020905+0000", 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4670000000000003e-06 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022364+0000", 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001459606 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022373+0000", 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5229999999999995e-06 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028305+0000", 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059320400000000004 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028340+0000", 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4603000000000001e-05 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=2, reqid=client.4562.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.459308707000002, 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0072410060000000004, 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024471+0000", 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.74e-07 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024570+0000", 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9729999999999996e-05 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024599+0000", 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8530999999999998e-05 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024605+0000", 2024-09-15T03:41:33.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2280000000000003e-06 2024-09-15T03:41:33.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024648+0000", 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2466000000000003e-05 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.024656+0000", 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8690000000000004e-06 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031688+0000", 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070326130000000001 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031712+0000", 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3394999999999999e-05 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=4, reqid=client.4562.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.458364469999999, 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063223339999999998, 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:33.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025415+0000", 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025416+0000", 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4600000000000004e-07 2024-09-15T03:41:33.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025430+0000", 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4636e-05 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025456+0000", 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5718e-05 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025462+0000", 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025538+0000", 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5916999999999998e-05 2024-09-15T03:41:33.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025548+0000", 2024-09-15T03:41:33.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9620000000000005e-06 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031716+0000", 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061672910000000001 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031737+0000", 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1569999999999998e-05 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.458064220000001, 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061423019999999997, 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025715+0000", 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025716+0000", 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5199999999999996e-07 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025913+0000", 2024-09-15T03:41:33.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000197252 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025939+0000", 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.552e-05 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025944+0000", 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7320000000000003e-06 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026100+0000", 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015575499999999999 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026109+0000", 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3700000000000001e-06 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031831+0000", 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005721271 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031857+0000", 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.675e-05 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=9, reqid=client.4562.0:52, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.447408156000002, 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059101470000000001, 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036371+0000", 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036372+0000", 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036625+0000", 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025348599999999998 2024-09-15T03:41:33.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036653+0000", 2024-09-15T03:41:33.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8116e-05 2024-09-15T03:41:33.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7190000000000007e-06 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036702+0000", 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0129000000000002e-05 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036711+0000", 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8769999999999999e-06 2024-09-15T03:41:33.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042262+0000", 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005551177 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042281+0000", 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9004e-05 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.459158446, 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067171740000000002, 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024621+0000", 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024622+0000", 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1489999999999999e-06 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024960+0000", 2024-09-15T03:41:33.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000337685 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.024999+0000", 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9569e-05 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025010+0000", 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0417e-05 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025062+0000", 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2043999999999998e-05 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.025074+0000", 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2227e-05 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031307+0000", 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062330859999999997 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031338+0000", 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0997000000000002e-05 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.45776854, 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079502919999999994, 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026011+0000", 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026099+0000", 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7769999999999995e-05 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026141+0000", 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2227000000000001e-05 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026149+0000", 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3730000000000004e-06 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026244+0000", 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5662999999999995e-05 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.026255+0000", 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0603e-05 2024-09-15T03:41:33.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033934+0000", 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076787699999999997 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033961+0000", 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7206999999999999e-05 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.456325385, 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0065285430000000004, 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027454+0000", 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027455+0000", 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9299999999999996e-07 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027493+0000", 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7716000000000003e-05 2024-09-15T03:41:33.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027532+0000", 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9904000000000003e-05 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027539+0000", 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027595+0000", 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5841000000000003e-05 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.027605+0000", 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7399999999999999e-06 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033967+0000", 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063620969999999997 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.033982+0000", 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5389000000000001e-05 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=64, reqid=client.4562.0:284, at_version=0'0, trim_to=0'0, roll_forward_to=45'28))", 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.445164031000001, 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090921019999999995, 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038616+0000", 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038629+0000", 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2891000000000001e-05 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038669+0000", 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9725000000000003e-05 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038682+0000", 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3095000000000001e-05 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038746+0000", 2024-09-15T03:41:33.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3850999999999996e-05 2024-09-15T03:41:33.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038758+0000", 2024-09-15T03:41:33.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2329e-05 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047686+0000", 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089280980000000006 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047707+0000", 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1089000000000001e-05 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:300 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~442368] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.474536194999999, 2024-09-15T03:41:33.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0389183930000001, 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009245+0000", 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.556e-06 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009830+0000", 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058575800000000003 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009844+0000", 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3247e-05 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009851+0000", 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4850000000000003e-06 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010438+0000", 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058667099999999998 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010441+0000", 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2119999999999999e-06 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031346+0000", 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0209051899999999 2024-09-15T03:41:33.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031477+0000", 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130712 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.032397+0000", 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091970699999999997 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037882+0000", 2024-09-15T03:41:33.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054851639999999998 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048077+0000", 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010194896 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048161+0000", 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4795000000000004e-05 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:301 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~720896] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.473911611999998, 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456429519999999, 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 301 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.553 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010594+0000", 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072566600000000003 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010605+0000", 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1228e-05 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010611+0000", 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8390000000000003e-06 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010679+0000", 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7966999999999995e-05 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010681+0000", 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1409999999999999e-06 2024-09-15T03:41:33.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033259+0000", 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.022577372 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033385+0000", 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012635200000000001 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.034673+0000", 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001287853 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038109+0000", 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003436508 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055462+0000", 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017353200999999999 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055511+0000", 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8217999999999998e-05 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:302 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~49152] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.473161230999999, 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.045100688, 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 302 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010619+0000", 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0699999999999996e-07 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010663+0000", 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552000000000003e-05 2024-09-15T03:41:33.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010680+0000", 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6175999999999999e-05 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010685+0000", 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8619999999999999e-06 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010726+0000", 2024-09-15T03:41:33.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0259000000000002e-05 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010727+0000", 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.753e-06 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035776+0000", 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0250488879999999 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035909+0000", 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013254900000000001 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.036628+0000", 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00071956899999999998 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038283+0000", 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016543490000000001 2024-09-15T03:41:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055650+0000", 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017367301000000002 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055719+0000", 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8622999999999997e-05 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:303 3.11s0 3:8f673fa9:::smithi10536496-24:head [setxattr _header (56) in=7b] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.473083344999999, 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0452317200000001, 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 303 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010697+0000", 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:41:33.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010700+0000", 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2509999999999999e-06 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010706+0000", 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4989999999999999e-06 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010711+0000", 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2250000000000002e-06 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010765+0000", 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4784000000000003e-05 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010767+0000", 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4160000000000001e-06 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037038+0000", 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.026271521 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037181+0000", 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014324800000000001 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037675+0000", 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049306399999999998 2024-09-15T03:41:33.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.039474+0000", 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017996729999999999 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055837+0000", 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016362393999999999 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055928+0000", 2024-09-15T03:41:33.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1051000000000006e-05 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:304 3.11s0 3:8f673fa9:::smithi10536496-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.473064423, 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.050398215, 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 304 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010717+0000", 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.125e-06 2024-09-15T03:41:33.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010724+0000", 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6880000000000002e-06 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010728+0000", 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.777e-06 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010795+0000", 2024-09-15T03:41:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7277e-05 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010796+0000", 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037918+0000", 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.02712114 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038025+0000", 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010774 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056315+0000", 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018289716000000001 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056417+0000", 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102336 2024-09-15T03:41:33.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061113+0000", 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046956749999999998 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:317 3.11s0 3:8b799747:::smithi10536496-30:head [rollback e] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.424698949, 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010406076, 2024-09-15T03:41:33.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 317 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059081+0000", 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.151e-06 2024-09-15T03:41:33.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059096+0000", 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4499999999999997e-06 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060096+0000", 2024-09-15T03:41:33.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010001019999999999 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060189+0000", 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2320999999999999e-05 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060639+0000", 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044999599999999998 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062024+0000", 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013859650000000001 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069431+0000", 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074061190000000001 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069486+0000", 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5831999999999997e-05 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.583 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:318 3.11s0 3:8b799747:::smithi10536496-30:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.424677913, 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010504207999999999, 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 318 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059102+0000", 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6600000000000002e-07 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059104+0000", 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.215e-06 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059110+0000", 2024-09-15T03:41:33.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0279999999999997e-06 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059114+0000", 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3340000000000003e-06 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061223+0000", 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002108249 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061279+0000", 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6418000000000001e-05 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061697+0000", 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000418109 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062208+0000", 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051137000000000003 2024-09-15T03:41:33.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069565+0000", 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073569799999999999 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069606+0000", 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0139000000000003e-05 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:374 3.11s0 3:8b799747:::smithi10536496-30:head [delete] snapc e=[c,b] ondisk+write+known_if_redirected+full_force+supports_pool_eio e51)", 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.453371545, 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086570359999999999, 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 374 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030410+0000", 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1500000000000002e-06 2024-09-15T03:41:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030418+0000", 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5989999999999996e-06 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030442+0000", 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4612000000000001e-05 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030450+0000", 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2959999999999999e-06 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030513+0000", 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030644+0000", 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130591 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.031027+0000", 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038316199999999999 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.031678+0000", 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00065064699999999997 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.039008+0000", 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073303159999999999 2024-09-15T03:41:33.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.039065+0000", 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6904000000000003e-05 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 56/51 ECSubWrite(tid=35, reqid=osd.7.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=56'16))", 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:23.072980+0000", 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.410799719, 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064563820000000001, 2024-09-15T03:41:33.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072980+0000", 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072980+0000", 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072981+0000", 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.64e-07 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.072993+0000", 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2063e-05 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073043+0000", 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0621999999999997e-05 2024-09-15T03:41:33.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073050+0000", 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1080000000000004e-06 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073116+0000", 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5690000000000003e-05 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.073129+0000", 2024-09-15T03:41:33.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2707999999999999e-05 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.079401+0000", 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062725139999999999 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:23.079436+0000", 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4712999999999998e-05 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.600 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.600 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:33.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:33.602 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:33.601+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:33.620 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:33.618+0000 7fd248430440 -1 osd.3 46 log_to_monitors true 2024-09-15T03:41:33.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:33.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:33.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:33.699 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:33.702 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:33.702+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:33.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:33.803 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:33.803+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:33.806 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:33.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:33.806 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:33.806 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:33.806 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:33.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:33.904 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:33.904+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:33.913 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:27 3.0s0 3:0304afdb:::smithi10536496-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.892937987, 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.028611145000000001, 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.914 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 27 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019130+0000", 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019131+0000", 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019132+0000", 2024-09-15T03:41:33.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7460000000000001e-06 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019139+0000", 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1720000000000004e-06 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019143+0000", 2024-09-15T03:41:33.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6720000000000003e-06 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023415+0000", 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042719689999999996 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023491+0000", 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5638999999999999e-05 2024-09-15T03:41:33.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034815+0000", 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011324555 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034893+0000", 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7682999999999995e-05 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047741+0000", 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01284832 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.890084791, 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:41:33.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:41:33.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.883477544000002, 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:41:33.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:41:33.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.883459930000001, 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:41:33.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:41:33.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.878734725000001, 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.942 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:64 3.0s0 3:0215fa29:::smithi10536496-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.866546145000001, 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.023494108999999999, 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 64 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045522+0000", 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045523+0000", 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.358e-06 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045894+0000", 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00037004799999999999 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045904+0000", 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456e-05 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045909+0000", 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6029999999999998e-06 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045938+0000", 2024-09-15T03:41:33.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9054e-05 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046014+0000", 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6334000000000005e-05 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046523+0000", 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000508903 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050383+0000", 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003860086 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068971+0000", 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018587939000000001 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.069016+0000", 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5327999999999997e-05 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.950 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.866152980999999, 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:41:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:41:33.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:41:33.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.866141439, 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:33.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:41:33.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:41:33.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.846949811999998, 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.966 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:41:33.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:41:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.846465917, 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:33.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:41:33.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.846424274, 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:41:33.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.844170826999999, 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:41:33.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.843182486, 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:41:33.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.994 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.786802135999999, 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036705979, 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:33.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125267+0000", 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125270+0000", 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161517+0000", 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036246426999999998 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161523+0000", 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161562+0000", 2024-09-15T03:41:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8571000000000001e-05 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161972+0000", 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041002299999999997 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:33.998 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.750700037000001, 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:33.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:41:34.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:41:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.746732373, 2024-09-15T03:41:34.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:34.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:287 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 8] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.883788784, 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.820123202, 2024-09-15T03:41:34.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 287 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028281+0000", 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5200000000000001e-06 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028286+0000", 2024-09-15T03:41:34.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028300+0000", 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3668e-05 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028307+0000", 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9539999999999998e-06 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029951+0000", 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001644419 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029955+0000", 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6260000000000001e-06 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037676+0000", 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077205920000000001 2024-09-15T03:41:34.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037910+0000", 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234212 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057184+0000", 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019273795999999999 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.059033+0000", 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00184902 2024-09-15T03:41:34.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848316+0000", 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78928352999999996 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848403+0000", 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6583999999999996e-05 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:288 3.10s0 3:092ceaee:::smithi10536496-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.883751842999999, 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030561617999999999, 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 288 2024-09-15T03:41:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028316+0000", 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028317+0000", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299999999999996e-07 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028320+0000", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0019999999999998e-06 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028328+0000", 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9379999999999992e-06 2024-09-15T03:41:34.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028357+0000", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8770999999999999e-05 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030005+0000", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016479140000000001 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.030007+0000", 2024-09-15T03:41:34.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5229999999999998e-06 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038132+0000", 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081245950000000001 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038195+0000", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2838999999999999e-05 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058661+0000", 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.020466254999999999 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058771+0000", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010993999999999999 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.058878+0000", 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107268 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 45/44 ECSubWriteReply(tid=39, last_complete=45'16, committed=1, applied=1))", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.847117425, 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78386887299999997, 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064956+0000", 2024-09-15T03:41:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848087+0000", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78313173400000002 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848099+0000", 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1173e-05 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848179+0000", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0921999999999995e-05 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848820+0000", 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064024899999999996 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:443 3.0s0 3:017cb3f5:::smithi10536496-29:head [copy-from ver 2] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.039800+0000", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.872268225, 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.023868127999999999, 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 443 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039800+0000", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039800+0000", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039802+0000", 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.114e-06 2024-09-15T03:41:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039811+0000", 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4719999999999995e-06 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039827+0000", 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.59e-05 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039833+0000", 2024-09-15T03:41:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.951e-06 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.039912+0000", 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8597999999999995e-05 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.040085+0000", 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017322400000000001 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049247+0000", 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0091619140000000002 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063180+0000", 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013932369999999999 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063605+0000", 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042501399999999998 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063668+0000", 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3571000000000005e-05 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:34.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:34.035 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:200: done (0 left) 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:201: rmattr oid 21 current snap is 15 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:202: rmattr oid 48 current snap is 15 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:202: done (1 left) 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:203: delete oid 31 current snap is 15 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:203: done (1 left) 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:204: delete oid 16 current snap is 15 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-15T03:41:34.036 INFO:tasks.rados.rados.0.smithi105.stdout:201: done (1 left) 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:204: done (0 left) 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:205: read oid 24 snap -1 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:205: expect (ObjNum 72 snap 7 seq_num 72) 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:206: read oid 50 snap -1 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:206: expect deleted 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:207: copy_from oid 49 from oid 11 current snap is 15 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:208: rmattr oid 7 current snap is 15 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:208: done (3 left) 2024-09-15T03:41:34.037 INFO:tasks.rados.rados.0.smithi105.stdout:209: delete oid 8 current snap is 15 2024-09-15T03:41:34.038 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:34.030+0000 7fd23ebee640 -1 osd.3 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:41:34.038 INFO:tasks.rados.rados.0.smithi105.stdout:206: done (3 left) 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:209: done (2 left) 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:210: copy_from oid 7 from oid 13 current snap is 15 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:211: snap_create 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:207: finishing copy_from to smithi10536496-49 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:207: got expected ENOENT (src dne) 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:207: finishing copy_from racing read to smithi10536496-49 2024-09-15T03:41:34.039 INFO:tasks.rados.rados.0.smithi105.stdout:205: expect (ObjNum 72 snap 7 seq_num 72) 2024-09-15T03:41:34.039 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:34.033+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:34.071 INFO:tasks.rados.rados.0.smithi105.stdout:210: finishing copy_from racing read to smithi10536496-7 2024-09-15T03:41:34.135 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:34.135+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:34.143 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:34.152 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:34.236 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:34.236+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:34.289 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:34.290 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:34.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:34.337 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:34.337+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:34.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:34.395 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:34.395 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:34.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:34.395 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:41:34.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.360613483000002, 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0090780140000000006, 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033039+0000", 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0100000000000005e-07 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:41:34.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5333e-05 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041513+0000", 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084585600000000004 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041517+0000", 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.318e-06 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041556+0000", 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8426000000000001e-05 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.041563+0000", 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6930000000000007e-06 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042090+0000", 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052617600000000005 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.042117+0000", 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6906999999999998e-05 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.346677325000002, 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.400 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:41:34.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:41:34.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:41:34.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:41:34.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:41:34.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:41:34.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:41:34.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:41:34.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.335048426, 2024-09-15T03:41:34.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:41:34.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:41:34.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:92 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:34.414 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.317430047999999, 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078039399999999997, 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 92 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:34.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076222+0000", 2024-09-15T03:41:34.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076223+0000", 2024-09-15T03:41:34.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:34.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076226+0000", 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6210000000000001e-06 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076234+0000", 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2740000000000002e-06 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076238+0000", 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4329999999999997e-06 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078113+0000", 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018749769999999999 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078219+0000", 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106016 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078626+0000", 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040713400000000002 2024-09-15T03:41:34.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.080519+0000", 2024-09-15T03:41:34.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001892712 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083957+0000", 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034382610000000002 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084026+0000", 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8678999999999998e-05 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.317404926000002, 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.423 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:41:34.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.307579177000001, 2024-09-15T03:41:34.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.432 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.278502144000001, 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.434 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:41:34.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:34.439 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.267585660999998, 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017309390000000001, 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126067+0000", 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126072+0000", 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3920000000000002e-06 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143252+0000", 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017179104000000001 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143257+0000", 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143315+0000", 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7958000000000003e-05 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143376+0000", 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0535000000000002e-05 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=19, reqid=client.4562.0:144, at_version=35'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.247200507999999, 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015149222, 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.443 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.85e-07 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146470+0000", 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7309999999999999e-05 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160975+0000", 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014505123 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160981+0000", 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161035+0000", 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3517999999999998e-05 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.065e-05 2024-09-15T03:41:34.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161571+0000", 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052504699999999995 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161601+0000", 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=23, reqid=client.4562.0:154, at_version=35'12, trim_to=0'0, roll_forward_to=35'11))", 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.228273111, 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.083287300999999994, 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165380+0000", 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0999999999999997e-07 2024-09-15T03:41:34.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165457+0000", 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6823999999999995e-05 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247921+0000", 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082464198000000002 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247926+0000", 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9880000000000004e-06 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247954+0000", 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8306e-05 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247960+0000", 2024-09-15T03:41:34.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7570000000000001e-06 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248650+0000", 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000689816 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248666+0000", 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6501999999999999e-05 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.452 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 45/44 ECSubWrite(tid=21, reqid=client.4562.0:290, at_version=45'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.363380831000001, 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077833549999999996, 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030271+0000", 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030272+0000", 2024-09-15T03:41:34.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2399999999999997e-07 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030290+0000", 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7683999999999999e-05 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030328+0000", 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8362999999999998e-05 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030335+0000", 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030374+0000", 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8415999999999999e-05 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030380+0000", 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7649999999999996e-06 2024-09-15T03:41:34.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038032+0000", 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076514269999999997 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.038055+0000", 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2776e-05 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s1 3.1f172733 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.38278802, 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017692614, 2024-09-15T03:41:34.459 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010866+0000", 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010929+0000", 2024-09-15T03:41:34.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010944+0000", 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4701e-05 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010952+0000", 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4610000000000002e-06 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011020+0000", 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8553000000000006e-05 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011023+0000", 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8150000000000002e-06 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028536+0000", 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017512913 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028557+0000", 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0716e-05 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:310 3.as1 3:5629793d:::smithi10536496-26:head [delete] snapc e=[e,c,b] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.388192901, 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0408854540000001, 2024-09-15T03:41:34.464 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 310 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:34.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005461+0000", 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.643e-06 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005465+0000", 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005480+0000", 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5407999999999999e-05 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005487+0000", 2024-09-15T03:41:34.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4229999999999999e-06 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008383+0000", 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002895309 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008386+0000", 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8949999999999998e-06 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.031743+0000", 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0233571130000001 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.032058+0000", 2024-09-15T03:41:34.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000315123 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.033215+0000", 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011571579999999999 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.037843+0000", 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046277380000000002 2024-09-15T03:41:34.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.045957+0000", 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081145560000000002 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046345+0000", 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000387229 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 50/49 ECSubWrite(tid=151, reqid=client.4562.0:335, at_version=50'20, trim_to=0'0, roll_forward_to=50'18))", 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.262880842000001, 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067600538000000002, 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130773+0000", 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:41:34.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130954+0000", 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018089000000000001 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196934+0000", 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065979530999999994 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196943+0000", 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197012+0000", 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8745999999999995e-05 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197025+0000", 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2909e-05 2024-09-15T03:41:34.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198352+0000", 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001327188 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198372+0000", 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9933999999999999e-05 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:347 3.ds0 3:baae61d4:::smithi10536496-31:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.245381044999998, 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089938310000000007, 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 347 2024-09-15T03:41:34.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148271+0000", 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.58e-07 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148275+0000", 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0269999999999997e-06 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148282+0000", 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.195e-06 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148286+0000", 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3669999999999999e-06 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.152643+0000", 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043572790000000004 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.152776+0000", 2024-09-15T03:41:34.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000133156 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.153340+0000", 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00056305400000000001 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154283+0000", 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00094319800000000002 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.156961+0000", 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026778100000000001 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157265+0000", 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030428699999999998 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.482 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:348 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.245361442, 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010390460000000001, 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 348 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.84e-07 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148293+0000", 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.858e-06 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148299+0000", 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9780000000000002e-06 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148303+0000", 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8249999999999998e-06 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154925+0000", 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066227459999999997 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.155002+0000", 2024-09-15T03:41:34.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6196e-05 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157677+0000", 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026754650000000001 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157794+0000", 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117167 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.158681+0000", 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088684099999999997 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.18s5 51/49 ECSubWrite(tid=153, reqid=client.4562.0:369, at_version=51'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:34.488 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.358589858999999, 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007414889, 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035062+0000", 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035063+0000", 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035480+0000", 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041686700000000002 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035525+0000", 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5948999999999997e-05 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035532+0000", 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1349999999999997e-06 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035566+0000", 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4785e-05 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.035574+0000", 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1099999999999997e-06 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.042460+0000", 2024-09-15T03:41:34.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068862660000000003 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.042477+0000", 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7286999999999999e-05 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 56/44 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=56'30, trim_to=0'0, roll_forward_to=51'28))", 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:34.493 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.319767431000001, 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096114640000000001, 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073885+0000", 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1100000000000005e-07 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073915+0000", 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9552000000000001e-05 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073949+0000", 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4022000000000001e-05 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.073955+0000", 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1349999999999997e-06 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074002+0000", 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7089000000000003e-05 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074013+0000", 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0446e-05 2024-09-15T03:41:34.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083471+0000", 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094584449999999994 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083496+0000", 2024-09-15T03:41:34.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4964e-05 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.5s4 56/44 ECSubWrite(tid=41, reqid=osd.15.0:41, at_version=56'32, trim_to=0'0, roll_forward_to=51'28))", 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:23.074553+0000", 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.319098613, 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092758249999999997, 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.499 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074553+0000", 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074553+0000", 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074554+0000", 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4799999999999997e-07 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074595+0000", 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1145e-05 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074631+0000", 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5735999999999998e-05 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.074638+0000", 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5100000000000004e-06 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.075409+0000", 2024-09-15T03:41:34.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00077193300000000004 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.075419+0000", 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9159999999999996e-06 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083796+0000", 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0083766959999999994 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:23.083829+0000", 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3141e-05 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.19s5 61/49 ECSubWrite(tid=25, reqid=client.4562.0:424, at_version=61'15, trim_to=0'0, roll_forward_to=60'12))", 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:28.112227+0000", 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.2814248949999998, 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0074112639999999999, 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112227+0000", 2024-09-15T03:41:34.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112227+0000", 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112228+0000", 2024-09-15T03:41:34.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.99e-07 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112412+0000", 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405099999999999 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112442+0000", 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9726e-05 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112448+0000", 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1149999999999999e-06 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112490+0000", 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1943000000000001e-05 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.112500+0000", 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9320000000000008e-06 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.119616+0000", 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007116921 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:41:34.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:28.119638+0000", 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1977e-05 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:41:34.510 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:34.511 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:34.512 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:34.511+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:34.513 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:34.513 INFO:tasks.rados.rados.0.smithi105.stdout:210: finishing copy_from to smithi10536496-7 2024-09-15T03:41:34.514 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 32 (ObjNum 69 snap 5 seq_num 69) dirty exists 2024-09-15T03:41:34.531 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:34.531 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:34.551 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:34.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:34.552 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-15T03:41:34.571 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:34.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:34.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:34.592 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:34.612 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:34.612 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:34.612 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:34.612 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:34.613 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:34.613 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:34.613 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:34.612+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:34.620 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:34.638 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:34.638 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:34.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:34.657 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:34.667 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:34.677 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:34.678 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:34.698 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:34.698 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:34.713 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:34.712+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:34.718 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:34.718 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:34.722 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:34.722 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:34.731 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:34.737 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:34.737 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:34.758 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:34.759 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:34.767 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:34.768 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:34.768 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:34.768 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:34.776 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-15T03:41:34.779 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:34.779 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:34.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:34.779+0000 7f3a9ffff640 0 -- 172.21.15.5:0/2726932333 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f3aa01934d0 msgr2=0x7f3aa01958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:34.815 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:34.814+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:34.865 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:34.865 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:34.873 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:34.916 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:34.915+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:34.971 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:34.971 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:34.971 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:34.972 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:34.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:35.000 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:35.000 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-15T03:41:35.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-15T03:41:35.017 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:35.016+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:35.019 INFO:tasks.rados.rados.0.smithi105.stdout:205: done (3 left) 2024-09-15T03:41:35.019 INFO:tasks.rados.rados.0.smithi105.stdout:207: done (2 left) 2024-09-15T03:41:35.019 INFO:tasks.rados.rados.0.smithi105.stdout:210: done (1 left) 2024-09-15T03:41:35.020 INFO:tasks.rados.rados.0.smithi105.stdout:211: done (0 left) 2024-09-15T03:41:35.020 INFO:tasks.rados.rados.0.smithi105.stdout:212: copy_from oid 8 from oid 4 current snap is 16 2024-09-15T03:41:35.020 INFO:tasks.rados.rados.0.smithi105.stdout:213: copy_from oid 2 from oid 47 current snap is 16 2024-09-15T03:41:35.020 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 39 current snap is 16 2024-09-15T03:41:35.020 INFO:tasks.rados.rados.0.smithi105.stdout:214: seq_num 101 ranges {638976=540672,1179648=507904,1687552=622592,2310144=32768} 2024-09-15T03:41:35.026 INFO:tasks.rados.rados.0.smithi105.stdout:214: writing smithi10536496-39 from 638976 to 1179648 tid 1 2024-09-15T03:41:35.030 INFO:tasks.rados.rados.0.smithi105.stdout:214: writing smithi10536496-39 from 1179648 to 1687552 tid 2 2024-09-15T03:41:35.034 INFO:tasks.rados.rados.0.smithi105.stdout:214: writing smithi10536496-39 from 1687552 to 2310144 tid 3 2024-09-15T03:41:35.034 INFO:tasks.rados.rados.0.smithi105.stdout:214: writing smithi10536496-39 from 2310144 to 2342912 tid 4 2024-09-15T03:41:35.034 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 36 current snap is 16 2024-09-15T03:41:35.034 INFO:tasks.rados.rados.0.smithi105.stdout:215: seq_num 102 ranges {1769472=557056,2326528=638976,2965504=524288} 2024-09-15T03:41:35.047 INFO:tasks.rados.rados.0.smithi105.stdout:215: writing smithi10536496-36 from 1769472 to 2326528 tid 1 2024-09-15T03:41:35.050 INFO:tasks.rados.rados.0.smithi105.stdout:215: writing smithi10536496-36 from 2326528 to 2965504 tid 2 2024-09-15T03:41:35.054 INFO:tasks.rados.rados.0.smithi105.stdout:215: writing smithi10536496-36 from 2965504 to 3489792 tid 3 2024-09-15T03:41:35.054 INFO:tasks.rados.rados.0.smithi105.stdout:216: snap_remove snap 15 2024-09-15T03:41:35.087 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:35.087 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:35.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:41:35.096 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:35.140 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:35.140+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:35.221 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:35.220+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:35.235 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:35.235 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:35.243 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:35.322 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:35.321+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:35.341 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:35.342 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:35.342 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:35.342 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:35.350 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:41:35.423 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:35.422+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:35.487 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:35.487 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:35.487 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:35.487 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:35.487 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:35.495 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:41:35.523 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:35.522+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:35.592 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:35.592 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:35.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:35.592 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-09-15T03:41:35.593 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:35.600 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:35.624 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:35.623+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:35.705 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:35.705 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:35.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:35.725 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:35.724+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:35.814 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:35.815 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:35.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:35.826 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:35.824+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:35.926 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:35.926 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:35.927 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:35.926+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:35.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:36.020 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:41:36.023 INFO:tasks.rados.rados.0.smithi105.stdout:215: finishing write tid 1 to smithi10536496-36 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:215: finishing write tid 2 to smithi10536496-36 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:215: finishing write tid 3 to smithi10536496-36 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:215: finishing write tid 4 to smithi10536496-36 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:215: finishing write tid 5 to smithi10536496-36 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 39 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:215: left oid 36 (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:215: done (4 left) 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:216: done (3 left) 2024-09-15T03:41:36.024 INFO:tasks.rados.rados.0.smithi105.stdout:217: snap_create 2024-09-15T03:41:36.028 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:36.028+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:36.045 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.045 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.055 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:36.129 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:36.128+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:36.197 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.197 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:36.230 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:36.229+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:36.308 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.308 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:36.331 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:36.330+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:36.418 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.418 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:36.432 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:36.432+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:36.528 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.528 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.534 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:36.533+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:36.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:36.635 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:36.634+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:36.638 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.638 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.647 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:36.736 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:36.735+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:36.748 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.749 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.757 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:36.837 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:36.835+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:36.858 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.858 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:36.938 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:36.937+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:36.967 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:36.968 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:36.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:37.004 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:37.022 INFO:tasks.rados.rados.0.smithi105.stdout:217: done (3 left) 2024-09-15T03:41:37.022 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 31 current snap is 17 2024-09-15T03:41:37.022 INFO:tasks.rados.rados.0.smithi105.stdout:218: seq_num 103 ranges {0=573440,573440=786432,1359872=376832} 2024-09-15T03:41:37.026 INFO:tasks.rados.rados.0.smithi105.stdout:218: writing smithi10536496-31 from 0 to 573440 tid 1 2024-09-15T03:41:37.031 INFO:tasks.rados.rados.0.smithi105.stdout:218: writing smithi10536496-31 from 573440 to 1359872 tid 2 2024-09-15T03:41:37.034 INFO:tasks.rados.rados.0.smithi105.stdout:218: writing smithi10536496-31 from 1359872 to 1736704 tid 3 2024-09-15T03:41:37.034 INFO:tasks.rados.rados.0.smithi105.stdout:219: read oid 25 snap -1 2024-09-15T03:41:37.034 INFO:tasks.rados.rados.0.smithi105.stdout:219: expect (ObjNum 97 snap 14 seq_num 97) 2024-09-15T03:41:37.034 INFO:tasks.rados.rados.0.smithi105.stdout:220: delete oid 18 current snap is 17 2024-09-15T03:41:37.037 INFO:tasks.rados.rados.0.smithi105.stdout:218: finishing write tid 1 to smithi10536496-31 2024-09-15T03:41:37.038 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:37.037+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:37.044 INFO:tasks.rados.rados.0.smithi105.stdout:220: done (5 left) 2024-09-15T03:41:37.044 INFO:tasks.rados.rados.0.smithi105.stdout:221: copy_from oid 30 from oid 17 current snap is 17 2024-09-15T03:41:37.044 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 11 current snap is 17 2024-09-15T03:41:37.044 INFO:tasks.rados.rados.0.smithi105.stdout:222: seq_num 105 ranges {0=606208,606208=688128,1294336=475136} 2024-09-15T03:41:37.047 INFO:tasks.rados.rados.0.smithi105.stdout:222: writing smithi10536496-11 from 0 to 606208 tid 1 2024-09-15T03:41:37.051 INFO:tasks.rados.rados.0.smithi105.stdout:222: writing smithi10536496-11 from 606208 to 1294336 tid 2 2024-09-15T03:41:37.054 INFO:tasks.rados.rados.0.smithi105.stdout:222: writing smithi10536496-11 from 1294336 to 1769472 tid 3 2024-09-15T03:41:37.054 INFO:tasks.rados.rados.0.smithi105.stdout:223: snap_remove snap 16 2024-09-15T03:41:37.106 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:37.106 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:37.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:37.139 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:37.138+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:37.155 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:37.155 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:37.177 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:37.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:37.197 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:37.197 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:37.217 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:37.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:37.237 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:37.240 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:37.238+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:37.257 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:37.257 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:37.259 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:37.260 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:37.260 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:37.260 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:37.260 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:37.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:37.277 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:37.277 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:37.298 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:37.298 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:37.317 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:37.317 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:37.337 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:37.337 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:37.340 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:37.340+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:37.358 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:37.372 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.353261474999996, 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094904819999999997, 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.23e-07 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017537+0000", 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8838000000000003e-05 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017912+0000", 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037478400000000002 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017919+0000", 2024-09-15T03:41:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8010000000000004e-06 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017987+0000", 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7138000000000007e-05 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017998+0000", 2024-09-15T03:41:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0908e-05 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026958+0000", 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089603229999999992 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026988+0000", 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0467000000000002e-05 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:5, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.352535416999999, 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085434570000000008, 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018225+0000", 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2619999999999999e-06 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018254+0000", 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8656000000000001e-05 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018297+0000", 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999999e-05 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018310+0000", 2024-09-15T03:41:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3052e-05 2024-09-15T03:41:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018392+0000", 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1527999999999996e-05 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4590000000000004e-06 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026715+0000", 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083134909999999992 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026767+0000", 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599000000000002e-05 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.351991147, 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080389840000000008, 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1199999999999999e-07 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018811+0000", 2024-09-15T03:41:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2938000000000001e-05 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018916+0000", 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010431 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018923+0000", 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019065+0000", 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014137500000000001 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019075+0000", 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0271e-05 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026774+0000", 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076987230000000002 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026807+0000", 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3655999999999998e-05 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.349151042999999, 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010778574000000001, 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021609+0000", 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021692+0000", 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999997e-05 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021735+0000", 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3581000000000002e-05 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021743+0000", 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3119999999999994e-06 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021822+0000", 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8437999999999997e-05 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021830+0000", 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2989999999999997e-06 2024-09-15T03:41:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032339+0000", 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010508768999999999 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032387+0000", 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7851e-05 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.348498259000003, 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010171065999999999, 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022262+0000", 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.043e-06 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022318+0000", 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6208000000000001e-05 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022353+0000", 2024-09-15T03:41:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4437000000000002e-05 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022362+0000", 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022408+0000", 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5775000000000001e-05 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022419+0000", 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1450999999999999e-05 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032351+0000", 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099319260000000006 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032432+0000", 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1032000000000006e-05 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.347830758999997, 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094003279999999995, 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022930+0000", 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022949+0000", 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9916e-05 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022995+0000", 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5173000000000003e-05 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023004+0000", 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2990000000000002e-06 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023067+0000", 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023073+0000", 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032300+0000", 2024-09-15T03:41:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092268320000000008 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032329+0000", 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8779999999999999e-05 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.347095052999997, 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087825990000000003, 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023665+0000", 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5199999999999995e-07 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023681+0000", 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-05 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023713+0000", 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1726000000000001e-05 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023720+0000", 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6830000000000002e-06 2024-09-15T03:41:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023765+0000", 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4988999999999999e-05 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023771+0000", 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9109999999999998e-06 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032437+0000", 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086659760000000006 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032447+0000", 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0222e-05 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.347045281, 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087050259999999994, 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023715+0000", 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-15T03:41:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023733+0000", 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8442999999999999e-05 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023770+0000", 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6585999999999998e-05 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023784+0000", 2024-09-15T03:41:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4440999999999999e-05 2024-09-15T03:41:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023829+0000", 2024-09-15T03:41:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5099000000000003e-05 2024-09-15T03:41:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023836+0000", 2024-09-15T03:41:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4400000000000002e-06 2024-09-15T03:41:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032396+0000", 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085599330000000005 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032419+0000", 2024-09-15T03:41:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3288000000000001e-05 2024-09-15T03:41:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.346466073999999, 2024-09-15T03:41:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081451809999999996, 2024-09-15T03:41:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024294+0000", 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024371+0000", 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6451000000000004e-05 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024415+0000", 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4292000000000002e-05 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024423+0000", 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0700000000000007e-06 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024479+0000", 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024488+0000", 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1109999999999997e-06 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032423+0000", 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079349969999999992 2024-09-15T03:41:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032438+0000", 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5279000000000001e-05 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.343618710000001, 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013181325000000001, 2024-09-15T03:41:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7000000000000004e-07 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027155+0000", 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027190+0000", 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5738000000000002e-05 2024-09-15T03:41:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027199+0000", 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1999999999999994e-06 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027257+0000", 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8690000000000002e-05 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027265+0000", 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8860000000000004e-06 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040256+0000", 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012990323999999999 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040322+0000", 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6392999999999999e-05 2024-09-15T03:41:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.343140005999999, 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012779128000000001, 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027620+0000", 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.32e-07 2024-09-15T03:41:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027692+0000", 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000006e-05 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027773+0000", 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0605000000000005e-05 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027782+0000", 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027884+0000", 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010173100000000001 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027894+0000", 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9210000000000004e-06 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040357+0000", 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463284999999999 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1026000000000003e-05 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.341852301999999, 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011382537, 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028908+0000", 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0300000000000001e-06 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028922+0000", 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4329e-05 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028975+0000", 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2573e-05 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028983+0000", 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-09-15T03:41:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029027+0000", 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3912000000000002e-05 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029036+0000", 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9560000000000003e-06 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040254+0000", 2024-09-15T03:41:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011217632999999999 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5880999999999997e-05 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.341839569000001, 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011534467, 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028921+0000", 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.057e-06 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028959+0000", 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8217999999999999e-05 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028993+0000", 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4425999999999999e-05 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029011+0000", 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7465e-05 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029270+0000", 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025923899999999998 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029280+0000", 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040431+0000", 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011151283 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040454+0000", 2024-09-15T03:41:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3309e-05 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.341201507999997, 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013707001, 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029559+0000", 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029676+0000", 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117273 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029720+0000", 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4026000000000002e-05 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029728+0000", 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3240000000000003e-06 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030022+0000", 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029338200000000002 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030037+0000", 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5082000000000001e-05 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043234+0000", 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013196548000000001 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043265+0000", 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1312000000000001e-05 2024-09-15T03:41:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.340141068000001, 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097244270000000008, 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030619+0000", 2024-09-15T03:41:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999998e-07 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030630+0000", 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030662+0000", 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1813000000000002e-05 2024-09-15T03:41:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030680+0000", 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8306999999999999e-05 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030722+0000", 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1823000000000003e-05 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030729+0000", 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040325+0000", 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095952650000000004 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040343+0000", 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8079000000000001e-05 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.339149566000003, 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011724098000000001, 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031742+0000", 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013129600000000001 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031779+0000", 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7311000000000002e-05 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031787+0000", 2024-09-15T03:41:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031842+0000", 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4617999999999998e-05 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031856+0000", 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.42e-05 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043313+0000", 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011457292000000001 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043334+0000", 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=9, reqid=client.4562.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.338951123999998, 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011489871, 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031809+0000", 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0130000000000001e-06 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031830+0000", 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0801999999999998e-05 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031864+0000", 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3575000000000003e-05 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031869+0000", 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5029999999999997e-06 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032118+0000", 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024897299999999999 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032128+0000", 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4879999999999996e-06 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043276+0000", 2024-09-15T03:41:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01114878 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043298+0000", 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1736999999999999e-05 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.338280511000001, 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010878669000000001, 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032507+0000", 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7140000000000001e-05 2024-09-15T03:41:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032542+0000", 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4986e-05 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032549+0000", 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2130000000000002e-06 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034852+0000", 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023031089999999998 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034862+0000", 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0156000000000001e-05 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043343+0000", 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084812489999999997 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043357+0000", 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4219000000000001e-05 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.335638478999996, 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008115901, 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035122+0000", 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035140+0000", 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7915999999999999e-05 2024-09-15T03:41:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035176+0000", 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6335999999999999e-05 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035184+0000", 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035238+0000", 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4048000000000002e-05 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035248+0000", 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043207+0000", 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079593690000000009 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043237+0000", 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9788e-05 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=20, reqid=client.4562.0:159, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.210340872000003, 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091519410000000002, 2024-09-15T03:41:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160419+0000", 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7499999999999999e-07 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160707+0000", 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028803500000000002 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160722+0000", 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5235000000000001e-05 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160727+0000", 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7600000000000002e-06 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160775+0000", 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7732999999999999e-05 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160785+0000", 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0380999999999999e-05 2024-09-15T03:41:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169551+0000", 2024-09-15T03:41:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087657610000000004 2024-09-15T03:41:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169570+0000", 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9261000000000001e-05 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.480 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:37.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:37.482 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:37.480+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:37.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:37.482 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:37.483 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:37.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:37.484 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:37.484 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:37.484 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:37.485 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:37.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:37.437+0000 7f77020fa640 0 -- 172.21.15.5:0/117032811 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f76fc198400 msgr2=0x7f76fc19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:37.582 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:37.581+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:37.626 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:37.626 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:37.626 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:37.626 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:37.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:37.659 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:37.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:37.684 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:37.683+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:37.736 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:37.736 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:37.736 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:37.736 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:37.737 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:37.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:37.785 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:37.784+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:1 3.8s0 3:136df151:::smithi10536496-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:37.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016470+0000", 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.829732210000003, 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010868803999999999, 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016470+0000", 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016470+0000", 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016473+0000", 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-09-15T03:41:37.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016566+0000", 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2530999999999999e-05 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016603+0000", 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7576000000000001e-05 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016610+0000", 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0659999999999998e-06 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016739+0000", 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012916100000000001 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016853+0000", 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113711 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017534+0000", 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068121200000000005 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019644+0000", 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002109625 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027270+0000", 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076264439999999996 2024-09-15T03:41:37.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027339+0000", 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8957999999999999e-05 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:2 3.8s0 3:136df151:::smithi10536496-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.829580073999999, 2024-09-15T03:41:37.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012099814, 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.99e-07 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016626+0000", 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016636+0000", 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8439999999999998e-06 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018474+0000", 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018381859999999999 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018568+0000", 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4424000000000003e-05 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019035+0000", 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046713100000000002 2024-09-15T03:41:37.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024240+0000", 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052042240000000003 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028633+0000", 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043929340000000002 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028722+0000", 2024-09-15T03:41:37.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9912000000000002e-05 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:3 3.8s0 3:136df151:::smithi10536496-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.829564220999998, 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016082863999999999, 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:37.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016639+0000", 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016640+0000", 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2729999999999999e-06 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.113e-06 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016650+0000", 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3489999999999999e-06 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019489+0000", 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028386459999999998 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019567+0000", 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7417999999999996e-05 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029120+0000", 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095539030000000007 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029205+0000", 2024-09-15T03:41:37.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4301000000000005e-05 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032721+0000", 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003516576 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.024276+0000", 2024-09-15T03:41:37.865 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.821926753, 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01044816, 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024276+0000", 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024276+0000", 2024-09-15T03:41:37.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024277+0000", 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2440000000000001e-06 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024364+0000", 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6335000000000006e-05 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024406+0000", 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2274999999999998e-05 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024414+0000", 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0120000000000005e-06 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024460+0000", 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5697e-05 2024-09-15T03:41:37.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024469+0000", 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2830000000000007e-06 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034693+0000", 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010224482 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034724+0000", 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0831999999999999e-05 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:61 3.8s0 3:12b3a6c7:::smithi10536496-21:head [append 0~344064,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043115+0000", 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.803087196, 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010576631, 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 61 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043115+0000", 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043115+0000", 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043117+0000", 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043673+0000", 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055560500000000003 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043688+0000", 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4741e-05 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043694+0000", 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4779999999999996e-06 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1158999999999997e-05 2024-09-15T03:41:37.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043831+0000", 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5313e-05 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.044497+0000", 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066600399999999999 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048631+0000", 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041346480000000003 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.053633+0000", 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050020159999999998 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.053692+0000", 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8955000000000001e-05 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:63 3.8s0 3:12b3a6c7:::smithi10536496-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.802470087000003, 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012980953, 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 63 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.72e-07 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8279999999999999e-06 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043740+0000", 2024-09-15T03:41:37.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9110000000000001e-06 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043742+0000", 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6460000000000002e-06 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045685+0000", 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001942706 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045767+0000", 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2405999999999997e-05 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054190+0000", 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084225670000000006 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054302+0000", 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112184 2024-09-15T03:41:37.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.056714+0000", 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024113329999999999 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:108 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.755385480999998, 2024-09-15T03:41:37.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010882141999999999, 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090818+0000", 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.39e-07 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090819+0000", 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8080000000000001e-06 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090824+0000", 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7419999999999997e-06 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090828+0000", 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7890000000000001e-06 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.093335+0000", 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025070370000000002 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.093382+0000", 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7268999999999998e-05 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096751+0000", 2024-09-15T03:41:37.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033692040000000002 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0794999999999998e-05 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.101699+0000", 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049071590000000003 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:286 3.8s0 3:136df151:::smithi10536496-1:head [create,setxattr dj[s][nsgcdlaO`ntnnogS`sZdZO[V\\]llWi\\gjj (11296) in=40b,setxattr qRNaVOjbUaQsk\\XsiZbOhrsOSoqX_Tqndtkb\\_Sa (1297) in=40b,setxattr `hdQPrsh[aUo`rPcbh\\tWT\\PatkS^p[P_Pdmcgdi (840) in=40b,setxattr RXicf\\^UcqO_mXh^am]mfcdttd\\mfOiPqZcomOo` (1822) in=40b,setxattr oZlXqWbdtOrc_U_m[apaV_Nr[hOOUgj`sVNpq^Rp (1473) in=40b,setxattr TrdTbcloOaiWXhjih\\mdXfYZcg\\^aOrnVOj^sf`s (10104) in=40b,setxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt (1263) in=40b,setxattr ^qaVseQnk[`qOXVa`m]]NtQ\\psXaNYand^tQYrWt (1051) in=40b,setxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q (418) in=40b,setxattr cfoPTk\\o\\Qqbi^XNOW[dbpcRb`OledY]YqcmmqUr (19016) in=40b,setxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd (10104) in=35b,setxattr mtVO^NRpeT]rQZYdTadneUO_^VXn^UdhZ^\\bUQiR (769) in=40b,setxattr prahkiNnm`U[rXSatop\\lk_p\\WVkdatg[XURZrsm (4560) in=40b,setxattr isZNUepUtPtRnYQaXpfelSVVR\\apSohRfqcfV`T_ (10033) in=40b,setxattr QZkproN[Zssrnra[rrbkaSXpatcreQpt`tWbOdoO (8349) in=40b,setxattr YqVPkaeiY^Tbe\\Rr\\heXkWjtOsi`iaN[TO\\^kP^] (876) in=40b,setxattr ZdfcZ]]snWVj]dZhZ]YSjVneRRoNbk`d^een^nqN (3368) in=40b,setxattr Y^XWcNfadqaYlbNpPbVRoPtgWreNtPrhWlfbbYY] (10804) in=40b,setxattr a`l_j\\^WlXY[ekSUURUhO (10104) in=21b,setxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ (1541) in=40b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.817969321, 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025921639, 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 286 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028235+0000", 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.872e-06 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028332+0000", 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028352+0000", 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0262999999999999e-05 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028359+0000", 2024-09-15T03:41:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4390000000000003e-06 2024-09-15T03:41:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029687+0000", 2024-09-15T03:41:37.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013271909999999999 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029881+0000", 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000194792 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.031152+0000", 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012703910000000001 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.033779+0000", 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002626724 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054105+0000", 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020326455 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054155+0000", 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0021000000000001e-05 2024-09-15T03:41:37.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=0, reqid=client.4562.0:300, at_version=50'10, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.810927684999999, 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012435007, 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:37.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6899999999999999e-07 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037696+0000", 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024206610000000002 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038322+0000", 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062559400000000004 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038330+0000", 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8789999999999992e-06 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038389+0000", 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8168999999999998e-05 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038400+0000", 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1083999999999999e-05 2024-09-15T03:41:37.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.047676+0000", 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092765030000000002 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.047710+0000", 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3847999999999998e-05 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=1, reqid=client.4562.0:301, at_version=50'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.808501924000002, 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011111194, 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:37.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0800000000000004e-07 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037845+0000", 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143296 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038740+0000", 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089520099999999998 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038747+0000", 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1459999999999999e-06 2024-09-15T03:41:37.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.039546+0000", 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000799082 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.039557+0000", 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048784+0000", 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092276149999999998 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048812+0000", 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7477000000000001e-05 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=2, reqid=client.4562.0:302, at_version=50'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.808354463000001, 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01098676, 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037849+0000", 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-07 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037903+0000", 2024-09-15T03:41:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4562999999999999e-05 2024-09-15T03:41:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038780+0000", 2024-09-15T03:41:37.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087622000000000004 2024-09-15T03:41:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038785+0000", 2024-09-15T03:41:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1189999999999996e-06 2024-09-15T03:41:37.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.044855+0000", 2024-09-15T03:41:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060705159999999998 2024-09-15T03:41:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.044866+0000", 2024-09-15T03:41:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0355e-05 2024-09-15T03:41:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048821+0000", 2024-09-15T03:41:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039556799999999996 2024-09-15T03:41:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048835+0000", 2024-09-15T03:41:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3805000000000001e-05 2024-09-15T03:41:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=3, reqid=client.4562.0:303, at_version=50'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.808296131999999, 2024-09-15T03:41:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011244417, 2024-09-15T03:41:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:37.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037921+0000", 2024-09-15T03:41:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3678e-05 2024-09-15T03:41:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038795+0000", 2024-09-15T03:41:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000874331 2024-09-15T03:41:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038799+0000", 2024-09-15T03:41:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3930000000000001e-06 2024-09-15T03:41:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045279+0000", 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064799879999999999 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045289+0000", 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3510000000000008e-06 2024-09-15T03:41:37.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048867+0000", 2024-09-15T03:41:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003578178 2024-09-15T03:41:37.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.049151+0000", 2024-09-15T03:41:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028403800000000002 2024-09-15T03:41:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:357 3.8s0 3:115f2b89:::smithi10536496-36:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:37.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.684303396000001, 2024-09-15T03:41:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012091859999999999, 2024-09-15T03:41:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 357 2024-09-15T03:41:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161901+0000", 2024-09-15T03:41:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5510000000000001e-06 2024-09-15T03:41:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161907+0000", 2024-09-15T03:41:37.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-09-15T03:41:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161921+0000", 2024-09-15T03:41:37.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.393e-05 2024-09-15T03:41:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161926+0000", 2024-09-15T03:41:37.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1309999999999997e-06 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161969+0000", 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2123000000000003e-05 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.162082+0000", 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011328999999999999 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.170186+0000", 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081040480000000008 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.171217+0000", 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010315389999999999 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.173948+0000", 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027302070000000001 2024-09-15T03:41:37.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.173991+0000", 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3572000000000002e-05 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.821275385, 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011226435999999999, 2024-09-15T03:41:37.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024928+0000", 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3099999999999996e-07 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024961+0000", 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2818000000000003e-05 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025047+0000", 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6197000000000002e-05 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025054+0000", 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3029999999999996e-06 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025108+0000", 2024-09-15T03:41:37.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4744000000000003e-05 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025120+0000", 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1212999999999999e-05 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.036125+0000", 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011005910000000001 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.036154+0000", 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.832e-05 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 51/49 ECSubWrite(tid=10, reqid=osd.10.0:10, at_version=51'19, trim_to=0'0, roll_forward_to=50'17))", 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:37.938 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.820897021, 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013366444, 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025307+0000", 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025333+0000", 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6886999999999999e-05 2024-09-15T03:41:37.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025376+0000", 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2821000000000002e-05 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025384+0000", 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025562+0000", 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017756799999999999 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025571+0000", 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-06 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038645+0000", 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013073817999999999 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038672+0000", 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7131000000000002e-05 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.943 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:438 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~753664] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8124952279999995, 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025886468999999999, 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 438 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033710+0000", 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4380000000000002e-06 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034822+0000", 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011121359999999999 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034849+0000", 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7406000000000002e-05 2024-09-15T03:41:37.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034859+0000", 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6830000000000003e-06 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034943+0000", 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4303999999999998e-05 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034953+0000", 2024-09-15T03:41:37.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5019999999999998e-06 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039409+0000", 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044563520000000002 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039523+0000", 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113514 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.041482+0000", 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019589410000000001 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048318+0000", 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006836033 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059547+0000", 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011228802 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059594+0000", 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7358000000000003e-05 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:439 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~720896] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8062236840000008, 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019746224, 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 439 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039981+0000", 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829999999999999e-06 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042101+0000", 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021201850000000001 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042116+0000", 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4994000000000001e-05 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042124+0000", 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7729999999999999e-06 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043391+0000", 2024-09-15T03:41:37.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012674959999999999 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043512+0000", 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120469 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.044649+0000", 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001137327 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048320+0000", 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036706320000000001 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059680+0000", 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011360093 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059725+0000", 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5472000000000001e-05 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:440 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~49152] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8040326229999994, 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02155733, 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 440 2024-09-15T03:41:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042171+0000", 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.39e-06 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042243+0000", 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1474e-05 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042256+0000", 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3016999999999999e-05 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042262+0000", 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4269999999999996e-06 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045435+0000", 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031722920000000002 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045540+0000", 2024-09-15T03:41:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105648 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046184+0000", 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064343199999999999 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049337+0000", 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003153089 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063687+0000", 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01435036 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063727+0000", 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0200999999999997e-05 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:441 3.8s0 3:115f2b89:::smithi10536496-36:head [setxattr _header (57) in=7b] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8039321909999995, 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021588698, 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 441 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5499999999999997e-07 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042275+0000", 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7900000000000001e-06 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042284+0000", 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8440000000000004e-06 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042294+0000", 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6569999999999992e-06 2024-09-15T03:41:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046742+0000", 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044484989999999999 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046840+0000", 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7651000000000004e-05 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047278+0000", 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043781499999999998 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049368+0000", 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020900620000000002 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063805+0000", 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014437324 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063859+0000", 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4301000000000001e-05 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:442 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8039038269999992, 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028656429000000001, 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 442 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042302+0000", 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4990000000000001e-06 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042309+0000", 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7139999999999996e-06 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042313+0000", 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2660000000000003e-06 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047492+0000", 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051795030000000002 2024-09-15T03:41:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047573+0000", 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0895000000000004e-05 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064140+0000", 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016566658000000001 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064227+0000", 2024-09-15T03:41:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199000000000003e-05 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.070955+0000", 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067281440000000001 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:37.972 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:37.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:37.974 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:37.973+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:38.030 INFO:tasks.rados.rados.0.smithi105.stdout:223: done (7 left) 2024-09-15T03:41:38.030 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 42 current snap is 17 2024-09-15T03:41:38.030 INFO:tasks.rados.rados.0.smithi105.stdout:224: seq_num 106 ranges {1769472=622592,2392064=442368,2834432=524288,3358720=196608} 2024-09-15T03:41:38.043 INFO:tasks.rados.rados.0.smithi105.stdout:224: writing smithi10536496-42 from 1769472 to 2392064 tid 1 2024-09-15T03:41:38.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:41:38.046 INFO:tasks.rados.rados.0.smithi105.stdout:224: writing smithi10536496-42 from 2392064 to 2834432 tid 2 2024-09-15T03:41:38.049 INFO:tasks.rados.rados.0.smithi105.stdout:224: writing smithi10536496-42 from 2834432 to 3358720 tid 3 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:224: writing smithi10536496-42 from 3358720 to 3555328 tid 4 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:225: delete oid 38 current snap is 17 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:218: finishing write tid 2 to smithi10536496-31 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:218: finishing write tid 3 to smithi10536496-31 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing write tid 1 to smithi10536496-39 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:218: finishing write tid 4 to smithi10536496-31 2024-09-15T03:41:38.051 INFO:tasks.rados.rados.0.smithi105.stdout:219: expect (ObjNum 97 snap 14 seq_num 97) 2024-09-15T03:41:38.074 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:38.073+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:38.097 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing write tid 2 to smithi10536496-39 2024-09-15T03:41:38.097 INFO:tasks.rados.rados.0.smithi105.stdout:218: finishing write tid 5 to smithi10536496-31 2024-09-15T03:41:38.097 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 23 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-15T03:41:38.097 INFO:tasks.rados.rados.0.smithi105.stdout:218: left oid 31 (ObjNum 103 snap 17 seq_num 103) 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing write tid 3 to smithi10536496-39 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:218: done (8 left) 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:219: done (7 left) 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:225: done (6 left) 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:226: rollback oid 29 current snap is 17 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:226: done (6 left) 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 24 current snap is 17 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing write tid 4 to smithi10536496-39 2024-09-15T03:41:38.098 INFO:tasks.rados.rados.0.smithi105.stdout:227: seq_num 108 ranges {1212416=655360,1867776=606208,2473984=475136,2949120=81920} 2024-09-15T03:41:38.105 INFO:tasks.rados.rados.0.smithi105.stdout:227: writing smithi10536496-24 from 1212416 to 1867776 tid 1 2024-09-15T03:41:38.108 INFO:tasks.rados.rados.0.smithi105.stdout:227: writing smithi10536496-24 from 1867776 to 2473984 tid 2 2024-09-15T03:41:38.111 INFO:tasks.rados.rados.0.smithi105.stdout:227: writing smithi10536496-24 from 2473984 to 2949120 tid 3 2024-09-15T03:41:38.112 INFO:tasks.rados.rados.0.smithi105.stdout:227: writing smithi10536496-24 from 2949120 to 3031040 tid 4 2024-09-15T03:41:38.112 INFO:tasks.rados.rados.0.smithi105.stdout:228: copy_from oid 37 from oid 48 current snap is 17 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing write tid 5 to smithi10536496-39 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:221: finishing copy_from racing read to smithi10536496-30 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:222: finishing write tid 1 to smithi10536496-11 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing write tid 6 to smithi10536496-39 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 41 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:214: left oid 39 (ObjNum 101 snap 16 seq_num 101) 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:221: finishing copy_from to smithi10536496-30 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 13 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:214: done (7 left) 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:221: done (6 left) 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:229: read oid 50 snap -1 2024-09-15T03:41:38.113 INFO:tasks.rados.rados.0.smithi105.stdout:229: expect deleted 2024-09-15T03:41:38.114 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 3 current snap is 17 2024-09-15T03:41:38.114 INFO:tasks.rados.rados.0.smithi105.stdout:222: finishing write tid 2 to smithi10536496-11 2024-09-15T03:41:38.114 INFO:tasks.rados.rados.0.smithi105.stdout:212: finishing copy_from racing read to smithi10536496-8 2024-09-15T03:41:38.114 INFO:tasks.rados.rados.0.smithi105.stdout:213: finishing copy_from racing read to smithi10536496-2 2024-09-15T03:41:38.114 INFO:tasks.rados.rados.0.smithi105.stdout:230: seq_num 109 ranges {1015808=671744,1687552=753664,2441216=409600} 2024-09-15T03:41:38.121 INFO:tasks.rados.rados.0.smithi105.stdout:230: writing smithi10536496-3 from 1015808 to 1687552 tid 1 2024-09-15T03:41:38.125 INFO:tasks.rados.rados.0.smithi105.stdout:230: writing smithi10536496-3 from 1687552 to 2441216 tid 2 2024-09-15T03:41:38.126 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:38.126 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:38.126 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:38.126 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:230: writing smithi10536496-3 from 2441216 to 2850816 tid 3 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:231: read oid 34 snap -1 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:231: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:232: rollback oid 19 current snap is 17 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:232: done (9 left) 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:233: read oid 36 snap -1 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:233: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:234: snap_create 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:222: finishing write tid 3 to smithi10536496-11 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:222: finishing write tid 4 to smithi10536496-11 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:212: finishing copy_from to smithi10536496-8 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 33 (ObjNum 85 snap 12 seq_num 85) dirty exists 2024-09-15T03:41:38.129 INFO:tasks.rados.rados.0.smithi105.stdout:222: finishing write tid 5 to smithi10536496-11 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 49 (ObjNum 105 snap 17 seq_num 105) dirty exists 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:222: left oid 11 (ObjNum 105 snap 17 seq_num 105) 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:213: finishing copy_from to smithi10536496-2 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 10 (ObjNum 61 snap 4 seq_num 61) dirty exists 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing write tid 1 to smithi10536496-42 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing write tid 2 to smithi10536496-42 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing write tid 3 to smithi10536496-42 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing write tid 4 to smithi10536496-42 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing write tid 1 to smithi10536496-24 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:228: finishing copy_from to smithi10536496-37 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:228: got expected ENOENT (src dne) 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:228: finishing copy_from racing read to smithi10536496-37 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing write tid 2 to smithi10536496-24 2024-09-15T03:41:38.130 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing write tid 3 to smithi10536496-24 2024-09-15T03:41:38.131 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing write tid 4 to smithi10536496-24 2024-09-15T03:41:38.131 INFO:tasks.rados.rados.0.smithi105.stdout:230: finishing write tid 1 to smithi10536496-3 2024-09-15T03:41:38.137 INFO:tasks.rados.rados.0.smithi105.stdout:230: finishing write tid 2 to smithi10536496-3 2024-09-15T03:41:38.137 INFO:tasks.rados.rados.0.smithi105.stdout:230: finishing write tid 3 to smithi10536496-3 2024-09-15T03:41:38.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:38.146 INFO:tasks.rados.rados.0.smithi105.stdout:233: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:38.184 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:38.183+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:38.241 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:38.242 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:38.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:38.283 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing write tid 5 to smithi10536496-42 2024-09-15T03:41:38.284 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing write tid 6 to smithi10536496-42 2024-09-15T03:41:38.284 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 28 (ObjNum 106 snap 17 seq_num 106) dirty exists 2024-09-15T03:41:38.284 INFO:tasks.rados.rados.0.smithi105.stdout:224: left oid 42 (ObjNum 106 snap 17 seq_num 106) 2024-09-15T03:41:38.291 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:38.290+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:38.378 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:38.376+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.393 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.362286042000001, 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011155541, 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029084+0000", 2024-09-15T03:41:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0603e-05 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029486+0000", 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000402144 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029493+0000", 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.165e-06 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029683+0000", 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000190478 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029706+0000", 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2849000000000001e-05 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040193+0000", 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010486822999999999 2024-09-15T03:41:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040228+0000", 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4993000000000002e-05 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:41, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.361047096, 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098373709999999993, 2024-09-15T03:41:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030312+0000", 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030333+0000", 2024-09-15T03:41:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0635000000000001e-05 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030374+0000", 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030383+0000", 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000004e-06 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030436+0000", 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2605000000000002e-05 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030447+0000", 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1576999999999999e-05 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040120+0000", 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096721959999999992 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040149+0000", 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9218e-05 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.360462386000002, 2024-09-15T03:41:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092836910000000002, 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030897+0000", 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6800000000000009e-07 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030911+0000", 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3589e-05 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030952+0000", 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1619000000000003e-05 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030962+0000", 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4679999999999998e-06 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031016+0000", 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3671999999999997e-05 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031025+0000", 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6180000000000001e-06 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040164+0000", 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091392020000000008 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040180+0000", 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5554999999999999e-05 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.359736400999999, 2024-09-15T03:41:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013955543000000001, 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031623+0000", 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031726+0000", 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103491 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031757+0000", 2024-09-15T03:41:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0908999999999998e-05 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031770+0000", 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2565000000000001e-05 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031863+0000", 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2622999999999997e-05 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031928+0000", 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5984999999999999e-05 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045547+0000", 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013618655 2024-09-15T03:41:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045578+0000", 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0565999999999998e-05 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.358773229999997, 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013036834000000001, 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032586+0000", 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032605+0000", 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-05 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032648+0000", 2024-09-15T03:41:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2763999999999999e-05 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032656+0000", 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032693+0000", 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6701e-05 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032699+0000", 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6849999999999998e-06 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045592+0000", 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012893345 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045622+0000", 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0182e-05 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.356135786000003, 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010405127, 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035224+0000", 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1800000000000005e-07 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035236+0000", 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1952e-05 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035267+0000", 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035273+0000", 2024-09-15T03:41:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035320+0000", 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6283999999999999e-05 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035326+0000", 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.105e-06 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045587+0000", 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010261562 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045628+0000", 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0469000000000002e-05 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.352001815000001, 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090014329999999997, 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039357+0000", 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039358+0000", 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039502+0000", 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000144272 2024-09-15T03:41:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039531+0000", 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8753999999999999e-05 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039537+0000", 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2009999999999996e-06 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039596+0000", 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9138000000000002e-05 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.039605+0000", 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048329+0000", 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087234679999999998 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.048358+0000", 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9442999999999999e-05 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=25, reqid=client.4562.0:101, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.307016881000003, 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011334317, 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084360+0000", 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7618999999999999e-05 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088093+0000", 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037333290000000001 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088098+0000", 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7010000000000004e-06 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088131+0000", 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2653000000000001e-05 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088136+0000", 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4979999999999997e-06 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095632+0000", 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074962550000000003 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095676+0000", 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3516999999999997e-05 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=27, reqid=client.4562.0:104, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.302522199999999, 2024-09-15T03:41:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098078109999999996, 2024-09-15T03:41:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088837+0000", 2024-09-15T03:41:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088853+0000", 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5770999999999999e-05 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095603+0000", 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067504870000000003 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095612+0000", 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4139999999999994e-06 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095655+0000", 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3417000000000001e-05 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095665+0000", 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0006e-05 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098619+0000", 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029542119999999999 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098644+0000", 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4975999999999999e-05 2024-09-15T03:41:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=32, to_read={3:edf86ef6:::smithi10536496-37:head=0,4096,0}, subchunks={3:edf86ef6:::smithi10536496-37:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.285918514999999, 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039281828999999997, 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105441+0000", 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:41:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105447+0000", 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5010000000000001e-06 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141752+0000", 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036305044000000002 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141762+0000", 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3940000000000002e-06 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141935+0000", 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017359499999999999 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.144722+0000", 2024-09-15T03:41:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00278671 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=34, reqid=client.4562.0:140, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.248845488000001, 2024-09-15T03:41:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038895118999999999, 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142514+0000", 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.449e-05 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180934+0000", 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038406005999999999 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180939+0000", 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4270000000000001e-06 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180984+0000", 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5751000000000003e-05 2024-09-15T03:41:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180995+0000", 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0332e-05 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181383+0000", 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038812500000000002 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181408+0000", 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5349000000000001e-05 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=39, reqid=client.4562.0:163, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.206357676000003, 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099943480000000001, 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185002+0000", 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185014+0000", 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.205e-05 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194530+0000", 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095162920000000008 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194535+0000", 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4090000000000002e-06 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194590+0000", 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4234000000000003e-05 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194595+0000", 2024-09-15T03:41:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4369999999999998e-06 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194977+0000", 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038248600000000002 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194995+0000", 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799e-05 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=40, reqid=client.4562.0:176, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.196858196000001, 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229890967, 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194501+0000", 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194514+0000", 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423889+0000", 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229374775 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423894+0000", 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0669999999999999e-06 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423938+0000", 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3967000000000001e-05 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423948+0000", 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2920000000000001e-06 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424370+0000", 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042234599999999998 2024-09-15T03:41:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424391+0000", 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1441999999999999e-05 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=74, reqid=client.4562.0:297, at_version=47'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.3790768, 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097576989999999999, 2024-09-15T03:41:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.29e-07 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012308+0000", 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6588e-05 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012483+0000", 2024-09-15T03:41:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017428699999999999 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012490+0000", 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.013059+0000", 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056855499999999999 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.013070+0000", 2024-09-15T03:41:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1286e-05 2024-09-15T03:41:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.022002+0000", 2024-09-15T03:41:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089322210000000006 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.022039+0000", 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7363999999999997e-05 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=75, reqid=client.4562.0:298, at_version=47'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.377096835, 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01362943, 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014263+0000", 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0100000000000001e-06 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014284+0000", 2024-09-15T03:41:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1742e-05 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014322+0000", 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7746000000000001e-05 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014330+0000", 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014378+0000", 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.799e-05 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014389+0000", 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027856+0000", 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013467549000000001 2024-09-15T03:41:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027891+0000", 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4925999999999997e-05 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=76, reqid=client.4562.0:299, at_version=47'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.376022162000002, 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01257076, 2024-09-15T03:41:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015337+0000", 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.65e-07 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015351+0000", 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4504e-05 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015447+0000", 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5996999999999996e-05 2024-09-15T03:41:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015454+0000", 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4250000000000003e-06 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016973+0000", 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001519248 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016980+0000", 2024-09-15T03:41:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4449999999999999e-06 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027896+0000", 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010915784 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027907+0000", 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0992000000000001e-05 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:399 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.339780698999999, 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014049582999999999, 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 399 2024-09-15T03:41:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051580+0000", 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-06 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051584+0000", 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3320000000000002e-06 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051594+0000", 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.027e-05 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051600+0000", 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3500000000000002e-06 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5226000000000003e-05 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051663+0000", 2024-09-15T03:41:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.25e-06 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058108+0000", 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064450269999999999 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058282+0000", 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017381800000000001 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059091+0000", 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080964600000000002 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062585+0000", 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003493875 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065538+0000", 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002952826 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065627+0000", 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339000000000006e-05 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:400 3.15s0 3:ac8f3d24:::smithi10536496-17:head [rollback c] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.339744179, 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016394477000000001, 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 400 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051615+0000", 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051618+0000", 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.977e-06 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051623+0000", 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5010000000000004e-06 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051626+0000", 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8009999999999999e-06 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051729+0000", 2024-09-15T03:41:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010270800000000001 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051734+0000", 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059522+0000", 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007788059 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059592+0000", 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0457999999999995e-05 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060158+0000", 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056606199999999999 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062892+0000", 2024-09-15T03:41:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002733769 2024-09-15T03:41:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.067964+0000", 2024-09-15T03:41:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050720879999999998 2024-09-15T03:41:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068009+0000", 2024-09-15T03:41:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4685999999999999e-05 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:401 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.339710486000001, 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016504504, 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 401 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0100000000000001e-07 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051651+0000", 2024-09-15T03:41:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-15T03:41:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051659+0000", 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:41:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051769+0000", 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011069 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051774+0000", 2024-09-15T03:41:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6430000000000003e-06 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060449+0000", 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008674747 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060522+0000", 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3393999999999996e-05 2024-09-15T03:41:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060841+0000", 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031853799999999999 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062921+0000", 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002080765 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068093+0000", 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051714500000000002 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068153+0000", 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9673000000000003e-05 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:41:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.3391929249999999, 2024-09-15T03:41:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37552997599999999, 2024-09-15T03:41:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-07 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.053528+0000", 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013622840000000001 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426011+0000", 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37248246099999999 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426019+0000", 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3089999999999996e-06 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426108+0000", 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9092999999999996e-05 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426118+0000", 2024-09-15T03:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0407e-05 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427667+0000", 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015489220000000001 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427696+0000", 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8337999999999999e-05 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.508 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:38.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:38.510 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:38.508+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:38.610 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:38.609+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:38.611 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:38.612 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:38.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:38.694 INFO:tasks.rados.rados.0.smithi105.stdout:230: finishing write tid 4 to smithi10536496-3 2024-09-15T03:41:38.695 INFO:tasks.rados.rados.0.smithi105.stdout:230: finishing write tid 5 to smithi10536496-3 2024-09-15T03:41:38.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 16 (ObjNum 109 snap 17 seq_num 109) dirty exists 2024-09-15T03:41:38.695 INFO:tasks.rados.rados.0.smithi105.stdout:230: left oid 3 (ObjNum 109 snap 17 seq_num 109) 2024-09-15T03:41:38.711 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:38.711+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:38.721 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:38.721 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:38.721 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:38.721 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:38.722 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:38.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:38.813 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:38.812+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:38.873 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:38.873 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.022241747999999, 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:38.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:41:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.022213643000001, 2024-09-15T03:41:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:41:38.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:41:38.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:38.886 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.727372537000001, 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:38.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:41:38.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:41:38.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=8, to_read={3:fff44eab:::smithi10536496-10:head=0,4096,0}, subchunks={3:fff44eab:::smithi10536496-10:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.837796193000003, 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046423195, 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033900+0000", 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033906+0000", 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-09-15T03:41:38.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080075+0000", 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046169038000000003 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080080+0000", 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080125+0000", 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5151000000000002e-05 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080322+0000", 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019686500000000001 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 35/32 ECSubWriteReply(tid=118, last_complete=35'2, committed=1, applied=1))", 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.784863911000002, 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021508349, 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:38.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086832+0000", 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086835+0000", 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0479999999999999e-06 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108275+0000", 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021439848000000001 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108283+0000", 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8609999999999998e-06 2024-09-15T03:41:38.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108320+0000", 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7330999999999999e-05 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108340+0000", 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9531999999999999e-05 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.763290544999997, 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041863826999999999, 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:38.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108419+0000", 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149106+0000", 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040686870999999999 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149110+0000", 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.335e-06 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149157+0000", 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7131e-05 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149164+0000", 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150245+0000", 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010804580000000001 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150269+0000", 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4128e-05 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:132 3.3s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.746481168000003, 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015465478, 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 132 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:38.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125215+0000", 2024-09-15T03:41:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:38.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125216+0000", 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7349999999999999e-06 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125223+0000", 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125227+0000", 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8840000000000004e-06 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129903+0000", 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046767349999999996 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129995+0000", 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1706000000000006e-05 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133187+0000", 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031916510000000002 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133234+0000", 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7862999999999999e-05 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140680+0000", 2024-09-15T03:41:38.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074451860000000003 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=25, to_read={3:fc38c57e:::smithi10536496-49:head=0,4096,0}, subchunks={3:fc38c57e:::smithi10536496-49:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.715240405999999, 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038157982, 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:38.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156460+0000", 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194316+0000", 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037855609999999998 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194320+0000", 2024-09-15T03:41:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999998e-06 2024-09-15T03:41:38.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194361+0000", 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0708999999999999e-05 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194613+0000", 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025227700000000002 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=31, to_read={3:f17f8228:::smithi10536496-40:head=0,167936,0}, subchunks={3:f17f8228:::smithi10536496-40:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.675792215999998, 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066290004, 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:38.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195904+0000", 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195911+0000", 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3030000000000002e-06 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262021+0000", 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066109629000000003 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262025+0000", 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5909999999999998e-06 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262085+0000", 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0183999999999998e-05 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262193+0000", 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107648 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:277 3.13s0 3:ca2bd2fa:::smithi10536496-45:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.842141177999999, 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0157117019999999, 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 277 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.921 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029556+0000", 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.542e-06 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029563+0000", 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2989999999999999e-06 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029577+0000", 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029601+0000", 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3200000000000001e-05 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029661+0000", 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0633e-05 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029665+0000", 2024-09-15T03:41:38.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5990000000000002e-06 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037509+0000", 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0078444609999999 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037649+0000", 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139476 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.045266+0000", 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076171729999999997 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.5s3 47/44 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=47'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.868519537000001, 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013250879, 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4600000000000001e-07 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003206+0000", 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9589000000000001e-05 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004002+0000", 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079689099999999998 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004013+0000", 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004302+0000", 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000288554 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004314+0000", 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2277e-05 2024-09-15T03:41:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016390+0000", 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012076 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016427+0000", 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6909000000000002e-05 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:305 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.867203303, 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0410521429999999, 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 305 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004494+0000", 2024-09-15T03:41:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1660000000000001e-06 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004502+0000", 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1540000000000002e-06 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004520+0000", 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7742000000000001e-05 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004530+0000", 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004681+0000", 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015092999999999999 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004692+0000", 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1232e-05 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009082+0000", 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043904649999999996 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009087+0000", 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7339999999999999e-06 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.033828+0000", 2024-09-15T03:41:38.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0247413160000001 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.034013+0000", 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018443999999999999 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045544+0000", 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011531351 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:306 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.867107124, 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0413215819999999, 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 306 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004590+0000", 2024-09-15T03:41:38.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3260000000000001e-06 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004596+0000", 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010000000000002e-06 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004613+0000", 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6892999999999999e-05 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004623+0000", 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268000000000001e-05 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009818+0000", 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051952880000000002 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009821+0000", 2024-09-15T03:41:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0189999999999998e-06 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035613+0000", 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.025791608 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035700+0000", 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7298000000000003e-05 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045910+0000", 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010209480999999999 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:307 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.867063210000001, 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.041622225, 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:38.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 307 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004633+0000", 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0200000000000001e-07 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004639+0000", 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5599999999999999e-06 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004649+0000", 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004656+0000", 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8870000000000003e-06 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:41:38.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051891749999999999 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035995+0000", 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026149322 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.036088+0000", 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3162e-05 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.046254+0000", 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010166405999999999 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:335 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:38.947 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.750895172, 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077783689000000003, 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 335 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120802+0000", 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120808+0000", 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2999999999999998e-06 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120821+0000", 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120826+0000", 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7879999999999997e-06 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120884+0000", 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8034e-05 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.121005+0000", 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120637 2024-09-15T03:41:38.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.130702+0000", 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096976079999999999 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.132697+0000", 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019945800000000001 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198536+0000", 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065839133999999994 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198584+0000", 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7796000000000001e-05 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:369 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~589824] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:38.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.846416794, 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022704654000000001, 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 369 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:41:38.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025281+0000", 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4159999999999998e-06 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028105+0000", 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028242380000000002 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028126+0000", 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0939999999999999e-05 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028134+0000", 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7649999999999993e-06 2024-09-15T03:41:38.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032973+0000", 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048395670000000003 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.033153+0000", 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017928700000000001 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034716+0000", 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001562873 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047854+0000", 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013138212999999999 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047923+0000", 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9306000000000004e-05 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047983+0000", 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0049000000000001e-05 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:370 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~573440] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.838731496000001, 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018633602999999999, 2024-09-15T03:41:38.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 370 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032966+0000", 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034471+0000", 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015050790000000001 2024-09-15T03:41:38.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034522+0000", 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0529999999999999e-05 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034528+0000", 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1750000000000002e-06 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041526+0000", 2024-09-15T03:41:38.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006997886 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041652+0000", 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126415 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.042320+0000", 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066831300000000002 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051454+0000", 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091331659999999999 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051523+0000", 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9029999999999995e-05 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051597+0000", 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4819000000000003e-05 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:371 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~196608] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.837157353999999, 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827797000000001, 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.965 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 371 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034539+0000", 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.116e-06 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034715+0000", 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017624599999999999 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034727+0000", 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1912999999999999e-05 2024-09-15T03:41:38.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034736+0000", 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044278+0000", 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095422289999999993 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044402+0000", 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124227 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.045119+0000", 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071647099999999999 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054221+0000", 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091019350000000002 2024-09-15T03:41:38.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054291+0000", 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0148999999999995e-05 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054366+0000", 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4883999999999997e-05 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:372 3.18s0 3:1c642a04:::smithi10536496-46:head [setxattr _header (56) in=7b] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.83695084, 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020478778, 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 372 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.971 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4199999999999995e-07 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034749+0000", 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034759+0000", 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4409999999999999e-06 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034769+0000", 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0064999999999999e-05 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047175+0000", 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012405657 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047305+0000", 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013068800000000001 2024-09-15T03:41:38.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047652+0000", 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034670499999999999 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054413+0000", 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067614540000000001 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055174+0000", 2024-09-15T03:41:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076077999999999996 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055223+0000", 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9076999999999998e-05 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:373 3.18s0 3:1c642a04:::smithi10536496-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:41:38.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.836920912, 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021752218, 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 373 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034775+0000", 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:41:38.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034778+0000", 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9249999999999999e-06 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034785+0000", 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5669999999999997e-06 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034790+0000", 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.51e-06 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048073+0000", 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013283497 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048128+0000", 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4455e-05 2024-09-15T03:41:38.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055532+0000", 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074043889999999999 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055612+0000", 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9971999999999999e-05 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:38.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.056527+0000", 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091438399999999997 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:38.984 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing write tid 5 to smithi10536496-24 2024-09-15T03:41:38.984 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing write tid 6 to smithi10536496-24 2024-09-15T03:41:38.984 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 31 (ObjNum 108 snap 17 seq_num 108) dirty exists 2024-09-15T03:41:38.984 INFO:tasks.rados.rados.0.smithi105.stdout:227: left oid 24 (ObjNum 108 snap 17 seq_num 108) 2024-09-15T03:41:38.984 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:38.982+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:38.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:39.004 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:41:39.005 INFO:tasks.rados.rados.0.smithi105.stdout:212: done (10 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:213: done (9 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:222: done (8 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:224: done (7 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:227: done (6 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:228: done (5 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:230: done (4 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:233: done (3 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:234: done (2 left) 2024-09-15T03:41:39.006 INFO:tasks.rados.rados.0.smithi105.stdout:235: snap_create 2024-09-15T03:41:39.084 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:39.083+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:39.101 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:39.101 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:39.185 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:39.184+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:39.208 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:39.209 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:39.285 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:39.285+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:5 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.297422052000002, 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010334054000000001, 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 5 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016634+0000", 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7000000000000004e-07 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016637+0000", 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6280000000000001e-06 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8289999999999997e-06 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016649+0000", 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017615+0000", 2024-09-15T03:41:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096610800000000003 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.017698+0000", 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3428000000000002e-05 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018129+0000", 2024-09-15T03:41:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043087300000000002 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022895+0000", 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047655759999999997 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026926+0000", 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040308160000000004 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026967+0000", 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1554999999999998e-05 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:6 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.297390149999998, 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012850047, 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:41:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6100000000000001e-07 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016667+0000", 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9769999999999999e-06 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016674+0000", 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-09-15T03:41:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016678+0000", 2024-09-15T03:41:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000003e-06 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018325+0000", 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001646385 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6032999999999995e-05 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027234+0000", 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088331369999999996 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027316+0000", 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1862999999999999e-05 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029515+0000", 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002199281 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.1s5 35/32 ECSubWrite(tid=6, reqid=client.4562.0:55, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.274796879999997, 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079303010000000007, 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039258+0000", 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1000000000000001e-07 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039586+0000", 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032722799999999999 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039615+0000", 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9383e-05 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039621+0000", 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1029999999999998e-06 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039673+0000", 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2261e-05 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039682+0000", 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7649999999999999e-06 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047166+0000", 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074841710000000004 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047189+0000", 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.228e-05 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:58 3.es0 3:7dd49206:::smithi10536496-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.273766361, 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0083733009999999997, 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040289+0000", 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1990000000000001e-06 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040639+0000", 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034862499999999998 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040646+0000", 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6290000000000001e-06 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040650+0000", 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.552e-06 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040691+0000", 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.07e-05 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040781+0000", 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0859000000000005e-05 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041560+0000", 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00077845500000000005 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044707+0000", 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031467969999999998 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048626+0000", 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039196150000000004 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.048662+0000", 2024-09-15T03:41:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.587e-05 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:59 3.es0 3:7dd49206:::smithi10536496-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.273398847999999, 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084161029999999994, 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 59 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040656+0000", 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040657+0000", 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4899999999999995e-07 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040659+0000", 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3319999999999999e-06 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040665+0000", 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6690000000000003e-06 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040668+0000", 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4630000000000001e-06 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.041990+0000", 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013215220000000001 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042074+0000", 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4060000000000005e-05 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042441+0000", 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000367432 2024-09-15T03:41:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047193+0000", 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047516440000000002 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049024+0000", 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001831378 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049072+0000", 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8053999999999998e-05 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:60 3.es0 3:7dd49206:::smithi10536496-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.273383058, 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010867740000000001, 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 60 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040673+0000", 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040675+0000", 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2479999999999999e-06 2024-09-15T03:41:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040681+0000", 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0170000000000002e-06 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040689+0000", 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6330000000000002e-06 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042594+0000", 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019041489999999999 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042676+0000", 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2633999999999995e-05 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049349+0000", 2024-09-15T03:41:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066727380000000001 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049419+0000", 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9868000000000003e-05 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.051540+0000", 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002121029 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:83 3.es0 3:7f98869d:::smithi10536496-28:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.251412139000003, 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080589019999999997, 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 83 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062643+0000", 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062645+0000", 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9199999999999998e-06 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062651+0000", 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1540000000000001e-06 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062654+0000", 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6729999999999998e-06 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.063750+0000", 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001095525 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.063801+0000", 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1573000000000002e-05 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064048+0000", 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024709100000000001 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064653+0000", 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060490800000000003 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070635+0000", 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059821070000000004 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070702+0000", 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6548e-05 2024-09-15T03:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:84 3.es0 3:7f98869d:::smithi10536496-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.251377925, 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011714999, 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 84 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062678+0000", 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062679+0000", 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6920000000000001e-06 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062685+0000", 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0979999999999999e-06 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062688+0000", 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3860000000000001e-06 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064229+0000", 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015411999999999999 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064308+0000", 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8621000000000005e-05 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070977+0000", 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006669538 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.071049+0000", 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1464000000000006e-05 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.074392+0000", 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003343586 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:152 3.es0 3:71615b7a:::smithi10536496-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.158442884999999, 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.031528529, 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 152 2024-09-15T03:41:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:41:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155613+0000", 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155616+0000", 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.88e-06 2024-09-15T03:41:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155623+0000", 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6950000000000003e-06 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155627+0000", 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999997e-06 2024-09-15T03:41:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156746+0000", 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001119229 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156828+0000", 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2294999999999997e-05 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157113+0000", 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028439999999999997 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.159322+0000", 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022090719999999999 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187068+0000", 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027746694999999998 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187141+0000", 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:153 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.158421535000002, 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.032862542000000002, 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 153 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:41:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155636+0000", 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7260000000000001e-06 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155642+0000", 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1449999999999996e-06 2024-09-15T03:41:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155645+0000", 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4470000000000002e-06 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157320+0000", 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016747769999999999 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157366+0000", 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5982999999999997e-05 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187194+0000", 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.029827955 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187240+0000", 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5494000000000002e-05 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188496+0000", 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012565250000000001 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:166 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.128536044000001, 2024-09-15T03:41:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.10104360799999999, 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 166 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185520+0000", 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4099999999999999e-07 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185522+0000", 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.588e-06 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185526+0000", 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5130000000000001e-06 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185528+0000", 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3099999999999999e-06 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186513+0000", 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098462300000000001 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8233000000000003e-05 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186810+0000", 2024-09-15T03:41:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023878899999999999 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187938+0000", 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001128698 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286460+0000", 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098521759 2024-09-15T03:41:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286563+0000", 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102654 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:167 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.128516771999998, 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.101989948, 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 167 2024-09-15T03:41:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:41:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185539+0000", 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8599999999999999e-07 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185540+0000", 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4020000000000001e-06 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185544+0000", 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6619999999999998e-06 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185547+0000", 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6330000000000002e-06 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188004+0000", 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024565619999999998 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188048+0000", 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3925999999999999e-05 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286831+0000", 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098782707999999997 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286925+0000", 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4609000000000002e-05 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.287528+0000", 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060316000000000005 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s2 45/44 ECSubWrite(tid=22, reqid=client.4562.0:286, at_version=45'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.031284+0000", 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.282771168, 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084639229999999999, 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031284+0000", 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031284+0000", 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031288+0000", 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5300000000000001e-06 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031727+0000", 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00043953200000000002 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031749+0000", 2024-09-15T03:41:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1617e-05 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031758+0000", 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7489999999999993e-06 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031808+0000", 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9103000000000001e-05 2024-09-15T03:41:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.031818+0000", 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0273e-05 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.039718+0000", 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079004350000000008 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.039748+0000", 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9683999999999999e-05 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.10s1 45/44 ECSubWrite(tid=39, reqid=client.4562.0:287, at_version=45'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.256925420999998, 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30713189699999999, 2024-09-15T03:41:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057131+0000", 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3799999999999996e-07 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057345+0000", 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021436799999999999 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362921+0000", 2024-09-15T03:41:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30557607399999998 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362927+0000", 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8549999999999998e-06 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362967+0000", 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9654999999999999e-05 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362974+0000", 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.364234+0000", 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012599200000000001 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.364262+0000", 2024-09-15T03:41:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7631e-05 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.es0 54/32 ECSubWriteReply(tid=55, last_complete=54'26, committed=1, applied=1))", 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.256385091999999, 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.195800472, 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057673+0000", 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.751e-06 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253340+0000", 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.19566710400000001 2024-09-15T03:41:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253347+0000", 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1250000000000004e-06 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253426+0000", 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8955999999999994e-05 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253471+0000", 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4121e-05 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s1 62/44 ECSubWrite(tid=41, reqid=client.4562.0:443, at_version=62'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.264902897000001, 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099932440000000001, 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049152+0000", 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.29e-07 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049191+0000", 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8832999999999999e-05 2024-09-15T03:41:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049237+0000", 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5318000000000002e-05 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049243+0000", 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.922e-06 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049303+0000", 2024-09-15T03:41:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0004999999999999e-05 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049314+0000", 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0779e-05 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059119+0000", 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098048660000000006 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059146+0000", 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6392000000000001e-05 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s2 62/44 ECSubWrite(tid=66, reqid=client.4562.0:441, at_version=0'0, trim_to=0'0, roll_forward_to=62'28))", 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.064007+0000", 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.25004815, 2024-09-15T03:41:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007908155, 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064007+0000", 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064007+0000", 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064008+0000", 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6700000000000003e-07 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064032+0000", 2024-09-15T03:41:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4142999999999999e-05 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064065+0000", 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3136000000000003e-05 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064071+0000", 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2899999999999999e-06 2024-09-15T03:41:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064119+0000", 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8092e-05 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.064130+0000", 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0665999999999999e-05 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.071847+0000", 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077167080000000001 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.071915+0000", 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8453000000000003e-05 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:546 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112325+0000", 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.2017302700000001, 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90424344599999995, 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 546 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112325+0000", 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112325+0000", 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112327+0000", 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.227e-06 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112339+0000", 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1459e-05 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112363+0000", 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4369999999999999e-05 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112370+0000", 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112424+0000", 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3996000000000003e-05 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112427+0000", 2024-09-15T03:41:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1e-06 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016449+0000", 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90402216700000004 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016568+0000", 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000119076 2024-09-15T03:41:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:547 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.201673437, 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90430617099999999, 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 547 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:41:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6700000000000003e-07 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112386+0000", 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112396+0000", 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0111e-05 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112399+0000", 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0199999999999999e-06 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112459+0000", 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9280999999999997e-05 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112461+0000", 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9530000000000002e-06 2024-09-15T03:41:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016597+0000", 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90413675000000004 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016688+0000", 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0605000000000004e-05 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:548 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.2016385439999999, 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90445608300000002, 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 548 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4000000000000002e-07 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112420+0000", 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7499999999999999e-06 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112427+0000", 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8809999999999998e-06 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112430+0000", 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999998e-06 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112476+0000", 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5525000000000002e-05 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112477+0000", 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-09-15T03:41:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016731+0000", 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.904254585 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016873+0000", 2024-09-15T03:41:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014156499999999999 2024-09-15T03:41:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.439 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.441 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:39.441 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:39.439+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:39.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:39.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.541 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:39.539+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:39.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:39.642 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:39.641+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:39.646 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:41:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:41:39.647 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.655 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:39.656 INFO:tasks.rados.rados.0.smithi105.stdout:231: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-15T03:41:39.671 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:39.742 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:39.741+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:39.755 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=0, reqid=client.4562.0:13, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.734419101, 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011510825000000001, 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019689+0000", 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5799999999999998e-07 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019738+0000", 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8652000000000001e-05 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019844+0000", 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106623 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019852+0000", 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019952+0000", 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100111 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019961+0000", 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:41:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031174+0000", 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011212744 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031199+0000", 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5278000000000001e-05 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.734357502999998, 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011454499999999999, 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019805+0000", 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5062000000000002e-05 2024-09-15T03:41:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020203+0000", 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039759399999999997 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020209+0000", 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6189999999999999e-06 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020263+0000", 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4410000000000003e-05 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020283+0000", 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0176000000000001e-05 2024-09-15T03:41:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031185+0000", 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010901569 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031204+0000", 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9663000000000001e-05 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=2, reqid=client.4562.0:14, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.733553026000003, 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010659454, 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020554+0000", 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020555+0000", 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-07 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020569+0000", 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3984e-05 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020610+0000", 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1411e-05 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020616+0000", 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1809999999999998e-06 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022402+0000", 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001785972 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022411+0000", 2024-09-15T03:41:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6920000000000008e-06 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031203+0000", 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087921389999999992 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031214+0000", 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0902e-05 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.732509643999997, 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012719667, 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021599+0000", 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:41:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021655+0000", 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6644999999999997e-05 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021703+0000", 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7549999999999997e-05 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021711+0000", 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1519999999999992e-06 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022571+0000", 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086020699999999999 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022577+0000", 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3339999999999998e-06 2024-09-15T03:41:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034283+0000", 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011705224 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034317+0000", 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.485e-05 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.731787214000001, 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012060876, 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022321+0000", 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023195+0000", 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087440700000000001 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023229+0000", 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3905000000000001e-05 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023237+0000", 2024-09-15T03:41:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.43e-06 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023278+0000", 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1137999999999998e-05 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023286+0000", 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5379999999999997e-06 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034356+0000", 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01107022 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034381+0000", 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5395999999999999e-05 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.730626217000001, 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010863246999999999, 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023481+0000", 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023482+0000", 2024-09-15T03:41:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023498+0000", 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6399e-05 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023536+0000", 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7944000000000001e-05 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023543+0000", 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.494e-06 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023580+0000", 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7379999999999998e-05 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023588+0000", 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3950000000000001e-06 2024-09-15T03:41:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034332+0000", 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010743068 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034344+0000", 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2918000000000001e-05 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.727774435000001, 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018217534000000001, 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026334+0000", 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8500000000000001e-07 2024-09-15T03:41:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026371+0000", 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6835999999999998e-05 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010326484 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036758+0000", 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0989999999999997e-05 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036813+0000", 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050000000000003e-05 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036822+0000", 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3149999999999998e-06 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044526+0000", 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077039439999999999 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.423e-05 2024-09-15T03:41:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.716942908, 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087714329999999993, 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-09-15T03:41:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037181+0000", 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5954999999999999e-05 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122876+0000", 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085695108000000006 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122883+0000", 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4069999999999998e-06 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122927+0000", 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552999999999998e-05 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122937+0000", 2024-09-15T03:41:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.006e-05 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124855+0000", 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001917687 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124879+0000", 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4002000000000002e-05 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:305 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.736242737000001, 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057908950000001, 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 305 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017866+0000", 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.468e-06 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017871+0000", 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8949999999999997e-06 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017882+0000", 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0781000000000001e-05 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017888+0000", 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7030000000000003e-06 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018332+0000", 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000444633 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018335+0000", 2024-09-15T03:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7690000000000001e-06 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023618+0000", 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.005282727 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023656+0000", 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7919000000000001e-05 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:306 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.736210727, 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057866579999999, 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 306 2024-09-15T03:41:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017900+0000", 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.756e-06 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017907+0000", 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017912+0000", 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4009999999999996e-06 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018381+0000", 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000468696 2024-09-15T03:41:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018383+0000", 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.906e-06 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023676+0000", 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0052937150000001 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023683+0000", 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0550000000000002e-06 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:307 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 25.736191107, 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.00579789, 2024-09-15T03:41:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 307 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017917+0000", 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017920+0000", 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3500000000000001e-06 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017926+0000", 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8529999999999997e-06 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017929+0000", 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3100000000000001e-06 2024-09-15T03:41:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018419+0000", 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048992199999999995 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018420+0000", 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.313e-06 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023709+0000", 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.005288537 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023714+0000", 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2310000000000004e-06 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s6 50/44 ECSubWrite(tid=31, reqid=client.4562.0:314, at_version=0'0, trim_to=0'0, roll_forward_to=50'20))", 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.679533688999999, 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011396837, 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074574+0000", 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-07 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.074585+0000", 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0251999999999999e-05 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083831+0000", 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092458939999999993 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083835+0000", 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5430000000000004e-06 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083884+0000", 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9052999999999999e-05 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.083894+0000", 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0246e-05 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.085938+0000", 2024-09-15T03:41:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020431540000000002 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.085971+0000", 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3030000000000001e-05 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s6 50/44 ECSubWrite(tid=35, reqid=client.4562.0:325, at_version=50'24, trim_to=0'0, roll_forward_to=50'20))", 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.667947633000001, 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075270610000000002, 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086179+0000", 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8609000000000001e-05 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160878+0000", 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074699163999999998 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160884+0000", 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6760000000000004e-06 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160935+0000", 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.117e-05 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160948+0000", 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3355e-05 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.161407+0000", 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045902099999999999 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.161430+0000", 2024-09-15T03:41:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2968000000000001e-05 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDScrubReserve(3.7s1 REQUEST e60)", 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.285083569999999, 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120973636, 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:41:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3400000000000001e-07 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469026+0000", 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3530000000000002e-06 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589898+0000", 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120871944 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589906+0000", 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5660000000000001e-06 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589955+0000", 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9027999999999999e-05 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589973+0000", 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8318e-05 2024-09-15T03:41:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589998+0000", 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4292999999999999e-05 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=62, reqid=client.4562.0:438, at_version=62'25, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:41:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.712620068, 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017875489000000001, 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041488+0000", 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042539+0000", 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001050823 2024-09-15T03:41:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042580+0000", 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0908000000000002e-05 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042587+0000", 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7929999999999998e-06 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042636+0000", 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8996999999999999e-05 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042645+0000", 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0899999999999994e-06 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059315+0000", 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016669970999999999 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059363+0000", 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7503999999999998e-05 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=63, reqid=client.4562.0:439, at_version=62'26, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.70840862, 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013689702, 2024-09-15T03:41:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045881+0000", 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018117900000000001 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045908+0000", 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7741000000000001e-05 2024-09-15T03:41:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045913+0000", 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0370000000000002e-06 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047115+0000", 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012019629999999999 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047126+0000", 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0210000000000001e-05 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059370+0000", 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01224459 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059389+0000", 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8328000000000001e-05 2024-09-15T03:41:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=64, reqid=client.4562.0:440, at_version=62'27, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.707805919, 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016955748, 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:41:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046339+0000", 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7045999999999997e-05 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046361+0000", 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2115000000000001e-05 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046366+0000", 2024-09-15T03:41:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2290000000000002e-06 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047844+0000", 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001477864 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047853+0000", 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6559999999999998e-06 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063237+0000", 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015383681 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063257+0000", 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0645999999999999e-05 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 62/32 ECSubWrite(tid=96, reqid=client.4562.0:446, at_version=0'0, trim_to=0'0, roll_forward_to=62'18))", 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.706603978, 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011862770999999999, 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047504+0000", 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5600000000000004e-07 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047516+0000", 2024-09-15T03:41:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1987999999999999e-05 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047611+0000", 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4741000000000007e-05 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047622+0000", 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1270999999999999e-05 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047672+0000", 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.99e-05 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047718+0000", 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5705999999999999e-05 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059338+0000", 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011619643000000001 2024-09-15T03:41:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059366+0000", 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8666e-05 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=65, reqid=client.4562.0:441, at_version=62'28, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.706227197, 2024-09-15T03:41:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015392342999999999, 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047881+0000", 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8700000000000004e-07 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047900+0000", 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8895000000000001e-05 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047932+0000", 2024-09-15T03:41:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1656999999999999e-05 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047939+0000", 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8329999999999996e-06 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051450+0000", 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035115120000000001 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051461+0000", 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1439e-05 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063261+0000", 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011799924999999999 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063273+0000", 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1195e-05 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 62/44 ECSubWrite(tid=41, reqid=client.4562.0:443, at_version=62'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.704823073, 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014072049999999999, 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049285+0000", 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049336+0000", 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1023000000000003e-05 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049377+0000", 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1591000000000002e-05 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049382+0000", 2024-09-15T03:41:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9609999999999996e-06 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051693+0000", 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023109720000000001 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051702+0000", 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9430000000000006e-06 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063337+0000", 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011634698000000001 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063356+0000", 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9599e-05 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:39.855 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.856 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [3] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15, 3] 2024-09-15T03:41:39.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:41:39.856 INFO:tasks.thrashosds.thrasher:inject_pause on osd.15 2024-09-15T03:41:39.856 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-15T03:41:39.856 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-15T03:41:39.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok config set filestore_inject_stall 3 2024-09-15T03:41:39.858 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:39.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:39.859 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:39.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:39.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:39.860 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:39.860 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:39.861 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:39.857+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:39.876 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:39.876 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:39.895 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:39.896 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:39.916 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:39.916 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:39.936 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:39.936 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:39.956 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:39.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:39.958 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:39.957+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:39.959 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:39.959 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:41:39.959 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:41:39.959 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:41:39.966 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-15T03:41:39.966 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:39.968 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:39.975 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:39.975 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:39.996 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:39.996 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:40.006 INFO:tasks.rados.rados.0.smithi105.stdout:229: done (2 left) 2024-09-15T03:41:40.006 INFO:tasks.rados.rados.0.smithi105.stdout:231: done (1 left) 2024-09-15T03:41:40.006 INFO:tasks.rados.rados.0.smithi105.stdout:235: done (0 left) 2024-09-15T03:41:40.006 INFO:tasks.rados.rados.0.smithi105.stdout:236: rollback oid 31 current snap is 19 2024-09-15T03:41:40.006 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 31 to 17 2024-09-15T03:41:40.006 INFO:tasks.rados.rados.0.smithi105.stdout:237: snap_create 2024-09-15T03:41:40.012 INFO:tasks.rados.rados.0.smithi105.stdout:236: finishing rollback tid 0 to smithi10536496-31 2024-09-15T03:41:40.015 INFO:tasks.rados.rados.0.smithi105.stdout:236: finishing rollback tid 1 to smithi10536496-31 2024-09-15T03:41:40.015 INFO:tasks.rados.rados.0.smithi105.stdout:236: finishing rollback tid 2 to smithi10536496-31 2024-09-15T03:41:40.015 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 32 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-15T03:41:40.021 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:40.022 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:40.041 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:40.042 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:40.059 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:40.058+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:40.061 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:40.081 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:40.081 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:40.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:40.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:40.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:40.100+0000 7f2afb7fe640 0 -- 172.21.15.5:0/3029071525 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f2afc1963c0 msgr2=0x7f2afc1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:40.107 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.108 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:40.161 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:40.160+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:40.212 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.213 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:40.265 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:40.263+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:40.319 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.319 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:40.323 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:40.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:40.362 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:40.362+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:40.463 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:40.461+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:40.467 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.468 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:40.564 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:40.562+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:40.614 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.614 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.622 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:40.665 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:40.664+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:40.727 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.727 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.736 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:40.766 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:40.765+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:40.834 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.834 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.843 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:40.867 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:40.866+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:40.939 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:40.939 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:40.948 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:40.968 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:40.967+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:41.015 INFO:tasks.rados.rados.0.smithi105.stdout:236: done (1 left) 2024-09-15T03:41:41.015 INFO:tasks.rados.rados.0.smithi105.stdout:237: done (0 left) 2024-09-15T03:41:41.015 INFO:tasks.rados.rados.0.smithi105.stdout:238: read oid 48 snap -1 2024-09-15T03:41:41.015 INFO:tasks.rados.rados.0.smithi105.stdout:238: expect deleted 2024-09-15T03:41:41.015 INFO:tasks.rados.rados.0.smithi105.stdout:239: delete oid 12 current snap is 20 2024-09-15T03:41:41.032 INFO:tasks.rados.rados.0.smithi105.stdout:238: done (1 left) 2024-09-15T03:41:41.032 INFO:tasks.rados.rados.0.smithi105.stdout:239: done (0 left) 2024-09-15T03:41:41.032 INFO:tasks.rados.rados.0.smithi105.stdout:240: setattr oid 19 current snap is 20 2024-09-15T03:41:41.033 INFO:tasks.rados.rados.0.smithi105.stdout:241: rollback oid 41 current snap is 20 2024-09-15T03:41:41.033 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 41 to 19 2024-09-15T03:41:41.033 INFO:tasks.rados.rados.0.smithi105.stdout:242: delete oid 13 current snap is 20 2024-09-15T03:41:41.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:41:41.038 INFO:tasks.rados.rados.0.smithi105.stdout:242: done (2 left) 2024-09-15T03:41:41.038 INFO:tasks.rados.rados.0.smithi105.stdout:243: read oid 38 snap -1 2024-09-15T03:41:41.038 INFO:tasks.rados.rados.0.smithi105.stdout:243: expect deleted 2024-09-15T03:41:41.038 INFO:tasks.rados.rados.0.smithi105.stdout:244: copy_from oid 2 from oid 36 current snap is 20 2024-09-15T03:41:41.038 INFO:tasks.rados.rados.0.smithi105.stdout:245: snap_create 2024-09-15T03:41:41.038 INFO:tasks.rados.rados.0.smithi105.stdout:241: finishing rollback tid 0 to smithi10536496-41 2024-09-15T03:41:41.039 INFO:tasks.rados.rados.0.smithi105.stdout:241: finishing rollback tid 1 to smithi10536496-41 2024-09-15T03:41:41.039 INFO:tasks.rados.rados.0.smithi105.stdout:241: finishing rollback tid 2 to smithi10536496-41 2024-09-15T03:41:41.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 53 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-15T03:41:41.044 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 12 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:41.049 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.050 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.059 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:41.059 INFO:tasks.rados.rados.0.smithi105.stdout:244: finishing copy_from racing read to smithi10536496-2 2024-09-15T03:41:41.066 INFO:tasks.rados.rados.0.smithi105.stdout:244: finishing copy_from to smithi10536496-2 2024-09-15T03:41:41.067 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 39 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-15T03:41:41.069 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:41.068+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:41.156 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.156 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.164 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:41.205 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:41.204+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:41.270 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:41.269+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:41.301 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.301 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.309 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:41.371 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:41.370+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:41.406 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.406 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.414 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:41.472 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:41.471+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:41.512 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.512 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.520 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:41.573 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:41.572+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:41.617 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.617 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.625 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:41.674 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:41.673+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:41.724 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.725 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.734 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:41:41.776 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:41.775+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:41.831 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.831 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.839 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:41:41.883 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:41.882+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:41.937 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:41.937 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:41.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:41.987 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:41.986+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:42.012 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:41:42.013 INFO:tasks.rados.rados.0.smithi105.stdout:240: done (4 left) 2024-09-15T03:41:42.013 INFO:tasks.rados.rados.0.smithi105.stdout:241: done (3 left) 2024-09-15T03:41:42.013 INFO:tasks.rados.rados.0.smithi105.stdout:243: done (2 left) 2024-09-15T03:41:42.013 INFO:tasks.rados.rados.0.smithi105.stdout:244: done (1 left) 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:245: done (0 left) 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:246: rollback oid 34 current snap is 21 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 34 to 17 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:247: copy_from oid 5 from oid 27 current snap is 21 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:248: read oid 30 snap -1 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:248: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:249: rollback oid 28 current snap is 21 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 28 to 17 2024-09-15T03:41:42.014 INFO:tasks.rados.rados.0.smithi105.stdout:250: delete oid 49 current snap is 21 2024-09-15T03:41:42.020 INFO:tasks.rados.rados.0.smithi105.stdout:246: finishing rollback tid 0 to smithi10536496-34 2024-09-15T03:41:42.020 INFO:tasks.rados.rados.0.smithi105.stdout:248: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:41:42.030 INFO:tasks.rados.rados.0.smithi105.stdout:249: finishing rollback tid 0 to smithi10536496-28 2024-09-15T03:41:42.030 INFO:tasks.rados.rados.0.smithi105.stdout:249: finishing rollback tid 1 to smithi10536496-28 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:248: done (4 left) 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:250: done (3 left) 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:251: read oid 29 snap -1 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:251: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:252: read oid 9 snap 18 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:252: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:253: read oid 16 snap -1 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:253: expect deleted 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:254: copy_from oid 10 from oid 41 current snap is 21 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:249: finishing rollback tid 2 to smithi10536496-28 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 55 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:246: finishing rollback tid 1 to smithi10536496-34 2024-09-15T03:41:42.031 INFO:tasks.rados.rados.0.smithi105.stdout:246: finishing rollback tid 2 to smithi10536496-34 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 35 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:246: done (6 left) 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:249: done (5 left) 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:255: read oid 2 snap 20 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:255: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:256: read oid 1 snap -1 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:256: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:41:42.032 INFO:tasks.rados.rados.0.smithi105.stdout:257: read oid 35 snap -1 2024-09-15T03:41:42.033 INFO:tasks.rados.rados.0.smithi105.stdout:257: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:41:42.033 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 17 current snap is 21 2024-09-15T03:41:42.033 INFO:tasks.rados.rados.0.smithi105.stdout:258: seq_num 113 ranges {278528=737280,1015808=557056,1572864=606208} 2024-09-15T03:41:42.035 INFO:tasks.rados.rados.0.smithi105.stdout:258: writing smithi10536496-17 from 278528 to 1015808 tid 1 2024-09-15T03:41:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:258: writing smithi10536496-17 from 1015808 to 1572864 tid 2 2024-09-15T03:41:42.043 INFO:tasks.rados.rados.0.smithi105.stdout:258: writing smithi10536496-17 from 1572864 to 2179072 tid 3 2024-09-15T03:41:42.043 INFO:tasks.rados.rados.0.smithi105.stdout:259: snap_create 2024-09-15T03:41:42.043 INFO:tasks.rados.rados.0.smithi105.stdout:247: finishing copy_from racing read to smithi10536496-5 2024-09-15T03:41:42.043 INFO:tasks.rados.rados.0.smithi105.stdout:257: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:41:42.050 INFO:tasks.rados.rados.0.smithi105.stdout:251: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T03:41:42.052 INFO:tasks.rados.rados.0.smithi105.stdout:256: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:41:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:252: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-15T03:41:42.062 INFO:tasks.rados.rados.0.smithi105.stdout:254: finishing copy_from racing read to smithi10536496-10 2024-09-15T03:41:42.062 INFO:tasks.rados.rados.0.smithi105.stdout:258: finishing write tid 1 to smithi10536496-17 2024-09-15T03:41:42.062 INFO:tasks.rados.rados.0.smithi105.stdout:258: finishing write tid 2 to smithi10536496-17 2024-09-15T03:41:42.062 INFO:tasks.rados.rados.0.smithi105.stdout:258: finishing write tid 3 to smithi10536496-17 2024-09-15T03:41:42.062 INFO:tasks.rados.rados.0.smithi105.stdout:255: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:42.072 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:42.072 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:42.078 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:42.077+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:42.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:42.180 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:42.179+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:42.180 INFO:tasks.rados.rados.0.smithi105.stdout:258: finishing write tid 4 to smithi10536496-17 2024-09-15T03:41:42.181 INFO:tasks.rados.rados.0.smithi105.stdout:258: finishing write tid 5 to smithi10536496-17 2024-09-15T03:41:42.181 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 24 (ObjNum 113 snap 21 seq_num 113) dirty exists 2024-09-15T03:41:42.181 INFO:tasks.rados.rados.0.smithi105.stdout:258: left oid 17 (ObjNum 113 snap 21 seq_num 113) 2024-09-15T03:41:42.181 INFO:tasks.rados.rados.0.smithi105.stdout:254: finishing copy_from to smithi10536496-10 2024-09-15T03:41:42.181 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 53 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-15T03:41:42.226 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:42.226 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:42.234 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:42.280 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:42.279+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:42.335 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:42.336 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:42.336 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:42.345 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:42.381 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:42.380+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:42.468 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:42.468 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:42.482 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:42.481+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:42.492 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:42.492 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:42.494 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:42.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:42.514 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:42.515 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:42.535 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:42.535 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:42.554 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:42.554 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:42.584 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:42.583+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:42.602 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:42.603 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:42.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:42.671 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:42.671 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:42.685 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:42.683+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:42.692 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:42.692 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:42.712 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:42.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:42.754 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:42.754 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:101 3.bs0 3:db73af7f:::smithi10536496-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.669775199, 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060658139999999999, 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:42.755 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 101 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:42.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082525+0000", 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082528+0000", 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0989999999999999e-06 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082540+0000", 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2673999999999999e-05 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082544+0000", 2024-09-15T03:41:42.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9559999999999999e-06 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083692+0000", 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011471949999999999 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083836+0000", 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014488899999999999 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084281+0000", 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044464500000000002 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085132+0000", 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00085125900000000004 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143106+0000", 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057973867999999998 2024-09-15T03:41:42.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143182+0000", 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5862e-05 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:102 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.669733589000003, 2024-09-15T03:41:42.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100342765, 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 102 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:42.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082569+0000", 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2279999999999998e-06 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082577+0000", 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9319999999999996e-06 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082584+0000", 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8009999999999998e-06 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084479+0000", 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018943709999999999 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084576+0000", 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7615000000000006e-05 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143227+0000", 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058651048999999997 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143302+0000", 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4547000000000003e-05 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182908+0000", 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039606548999999998 2024-09-15T03:41:42.766 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=25, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.666641415000001, 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057539988, 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:42.767 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.98e-07 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085660+0000", 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.902e-06 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142986+0000", 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057326329000000002 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142992+0000", 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143038+0000", 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6576999999999998e-05 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143198+0000", 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015933200000000001 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:111 3.7s0 3:edf86ef6:::smithi10536496-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.657918141000003, 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050567135999999999, 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 111 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.772 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094382+0000", 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094384+0000", 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7489999999999999e-06 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094393+0000", 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2500000000000006e-06 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094397+0000", 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8040000000000001e-06 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002394565 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096872+0000", 2024-09-15T03:41:42.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0492000000000001e-05 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105071+0000", 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081983339999999998 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105164+0000", 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3202999999999995e-05 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.144948+0000", 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039784183000000001 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:141 3.bs0 3:dc608269:::smithi10536496-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:42.778 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.612295109000002, 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055455817999999997, 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 141 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6600000000000003e-07 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140006+0000", 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5409999999999999e-06 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140011+0000", 2024-09-15T03:41:42.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.843e-06 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140013+0000", 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.621e-06 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:42.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002515181 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142605+0000", 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6328999999999994e-05 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182018+0000", 2024-09-15T03:41:42.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039413493000000001 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182116+0000", 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8159000000000007e-05 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.195460+0000", 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013343384999999999 2024-09-15T03:41:42.784 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:273 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~802816] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 31.725381065000001, 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0316972600000001, 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.785 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:42.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026920+0000", 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5629999999999999e-06 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027741+0000", 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082160000000000002 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027764+0000", 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2495000000000001e-05 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027772+0000", 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027837+0000", 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5062999999999997e-05 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027841+0000", 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:41:42.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043211+0000", 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.015370181 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043389+0000", 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017807500000000001 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044733+0000", 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013438510000000001 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048847+0000", 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041134250000000004 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058556+0000", 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097088929999999997 2024-09-15T03:41:42.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058615+0000", 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9623999999999997e-05 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:274 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~376832] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:42.792 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 31.722900989999999, 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.029738034, 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 274 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:42.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029399+0000", 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6099999999999999e-07 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029738+0000", 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00033941400000000001 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029756+0000", 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524e-05 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029763+0000", 2024-09-15T03:41:42.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9609999999999999e-06 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029831+0000", 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8169000000000004e-05 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029833+0000", 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045818+0000", 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0159847550000001 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045920+0000", 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102085 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.046687+0000", 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00076710099999999996 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049023+0000", 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023356919999999999 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:42.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059069+0000", 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010046019999999999 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059136+0000", 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7156999999999994e-05 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:275 3.17s0 3:ec08d906:::smithi10536496-14:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 31.722509106, 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0294973439999999, 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.800 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 275 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:42.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029791+0000", 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029807+0000", 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6107e-05 2024-09-15T03:41:42.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029828+0000", 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1475999999999999e-05 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029838+0000", 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9879999999999999e-06 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029885+0000", 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6887e-05 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029887+0000", 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5030000000000001e-06 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047307+0000", 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0174202429999999 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047434+0000", 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012744699999999999 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048033+0000", 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00059865199999999997 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049643+0000", 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016096159999999999 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059241+0000", 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095980579999999996 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059287+0000", 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6726e-05 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:276 3.17s0 3:ec08d906:::smithi10536496-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 31.722442545, 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0302367509999999, 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.808 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 276 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029862+0000", 2024-09-15T03:41:42.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4259999999999996e-06 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029875+0000", 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3132999999999999e-05 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029880+0000", 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029913+0000", 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3079e-05 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029914+0000", 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.164e-06 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048304+0000", 2024-09-15T03:41:42.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.018390516 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048405+0000", 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010111300000000001 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059337+0000", 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010931523 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059408+0000", 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0962000000000003e-05 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.060093+0000", 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068532499999999997 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.814 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.751620623000001, 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.030189832, 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000680+0000", 2024-09-15T03:41:42.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000683+0000", 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6610000000000002e-06 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000708+0000", 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5165999999999999e-05 2024-09-15T03:41:42.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000714+0000", 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6250000000000004e-06 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005624+0000", 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049095390000000001 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005627+0000", 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-06 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030812+0000", 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025185383 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030868+0000", 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6032000000000002e-05 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.819 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.751565054, 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301912289999999, 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.820 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000735+0000", 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000739+0000", 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0119999999999999e-06 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000747+0000", 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1159999999999999e-06 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000751+0000", 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048937850000000003 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5700000000000002e-07 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030914+0000", 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252686520000001 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030925+0000", 2024-09-15T03:41:42.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1248e-05 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.751456652999998, 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301360509999999, 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.825 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000843+0000", 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000856+0000", 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2296999999999999e-05 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000869+0000", 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3045e-05 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000878+0000", 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6749999999999997e-06 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005662+0000", 2024-09-15T03:41:42.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047836379999999998 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005663+0000", 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3300000000000002e-07 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030967+0000", 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025303984 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030978+0000", 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1972e-05 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.4s2 REQUEST e53)", 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.976026966999999, 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27094768699999999, 2024-09-15T03:41:42.830 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776274+0000", 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.007e-06 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776283+0000", 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9870000000000005e-06 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047099+0000", 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.270815584 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047106+0000", 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7249999999999997e-06 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047165+0000", 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8955000000000001e-05 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047186+0000", 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047220+0000", 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4156999999999998e-05 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.834 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.10s3 REQUEST e61)", 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.039650608000001, 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.318056805, 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712649+0000", 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4e-07 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712654+0000", 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5340000000000001e-06 2024-09-15T03:41:42.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030357+0000", 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.31770321000000001 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030364+0000", 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.494e-06 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030652+0000", 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000288733 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030676+0000", 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4046e-05 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030705+0000", 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8847999999999999e-05 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.838 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "replica_scrub(pg: 3.10s3,from:0'0,to:58'33,epoch:62/49,start:3:08000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.719398177, 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072588579, 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032902+0000", 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7899999999999997e-07 2024-09-15T03:41:42.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032909+0000", 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4200000000000001e-06 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105400+0000", 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072491085999999996 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105407+0000", 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6800000000000004e-06 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105450+0000", 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2812999999999998e-05 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105459+0000", 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4830000000000005e-06 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105489+0000", 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0218000000000001e-05 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:494 3.bs0 3:d31b706c:::smithi10536496-7:head [copy-from ver 32] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.7184745770000003, 2024-09-15T03:41:42.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.39424540800000002, 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 494 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033827+0000", 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3460000000000001e-06 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033834+0000", 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5850000000000002e-06 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033850+0000", 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5528000000000001e-05 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033857+0000", 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4819999999999997e-06 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033931+0000", 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3873000000000002e-05 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.034026+0000", 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4959999999999999e-05 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.052129+0000", 2024-09-15T03:41:42.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018102593 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.054670+0000", 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025406600000000001 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.428003+0000", 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.37333323699999998 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.428070+0000", 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7144000000000006e-05 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPushReply(3.5s2 71/69 [PushReplyOp(3:a7103f89:::smithi10536496-39:head)])", 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.6740122959999999, 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.38581299400000002, 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:42.849 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:42.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.02e-07 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078289+0000", 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.311e-06 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463477+0000", 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.38518750299999999 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463488+0000", 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1687e-05 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463605+0000", 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011645 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.464100+0000", 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00049494100000000004 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.852 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:554 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.6239549640000002, 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5249715749999999, 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.853 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 554 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128346+0000", 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3520000000000001e-06 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128356+0000", 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.876e-06 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128385+0000", 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9113000000000001e-05 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128394+0000", 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7099999999999996e-06 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128458+0000", 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4274000000000002e-05 2024-09-15T03:41:42.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128463+0000", 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6240000000000001e-06 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.649581+0000", 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.521117751 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.649784+0000", 2024-09-15T03:41:42.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000202759 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.653316+0000", 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035321160000000001 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:555 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:42.858 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.6238895739999997, 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.525273152, 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 555 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:42.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9699999999999996e-07 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128414+0000", 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.912e-06 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128423+0000", 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3740000000000004e-06 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128428+0000", 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7840000000000003e-06 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128504+0000", 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5908999999999994e-05 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128505+0000", 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.204e-06 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650194+0000", 2024-09-15T03:41:42.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5216883160000001 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650271+0000", 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7257999999999998e-05 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.653683+0000", 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003411898 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.863 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:556 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.6238490389999996, 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5256133200000002, 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 556 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.864 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:42.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128451+0000", 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128455+0000", 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0350000000000003e-06 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128524+0000", 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9486000000000005e-05 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128531+0000", 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6089999999999997e-06 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128573+0000", 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1913999999999999e-05 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128574+0000", 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.155e-06 2024-09-15T03:41:42.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650502+0000", 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5219284549999998 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650630+0000", 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012748800000000001 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:42.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.654063+0000", 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034336480000000001 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:42.869 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:42.870 INFO:tasks.rados.rados.0.smithi105.stdout:247: finishing copy_from to smithi10536496-5 2024-09-15T03:41:42.870 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:41:42.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:42.872 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:42.869+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:42.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:42.935 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:42.955 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:42.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:42.971 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:42.971+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:42.975 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:42.975 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:42.996 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:42.996 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:247: done (9 left) 2024-09-15T03:41:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:251: done (8 left) 2024-09-15T03:41:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:252: done (7 left) 2024-09-15T03:41:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:253: done (6 left) 2024-09-15T03:41:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:254: done (5 left) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:255: done (4 left) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:256: done (3 left) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:257: done (2 left) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:258: done (1 left) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:259: done (0 left) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:260: read oid 23 snap 21 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:260: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:261: setattr oid 36 current snap is 22 2024-09-15T03:41:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 48 current snap is 22 2024-09-15T03:41:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:262: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2024-09-15T03:41:43.017 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:43.017 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:43.017 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:43.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:43.017 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:43.017 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:43.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:43.016+0000 7f5f2affd640 0 -- 172.21.15.5:0/1462201741 >> [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] conn(0x7f5f2c18f780 msgr2=0x7f5f2c191b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:262: writing smithi10536496-48 from 0 to 753664 tid 1 2024-09-15T03:41:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:262: writing smithi10536496-48 from 753664 to 1441792 tid 2 2024-09-15T03:41:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:262: writing smithi10536496-48 from 1441792 to 1916928 tid 3 2024-09-15T03:41:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:263: snap_remove snap 17 2024-09-15T03:41:43.029 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:43.072 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:43.071+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:43.172 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:43.172 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:43.172 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:43.172 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:43.172 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:43.173 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:43.171+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:43.180 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:43.238 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:43.239 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:43.259 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:43.259 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:43.274 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:43.272+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:43.279 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:43.279 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=77, reqid=client.4562.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=47'19))", 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.254549404999999, 2024-09-15T03:41:43.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92662449099999999, 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028438+0000", 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2099999999999996e-07 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028447+0000", 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8759999999999994e-06 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954390+0000", 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92594294600000004 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954412+0000", 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2218e-05 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954469+0000", 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7000999999999999e-05 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954479+0000", 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5489999999999995e-06 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955037+0000", 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055842499999999998 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955062+0000", 2024-09-15T03:41:43.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4754999999999999e-05 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:310 3.as0 3.bc9e946a (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e48)", 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.272398275, 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99312189699999998, 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.290 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 310 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:43.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010590+0000", 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.581e-06 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010594+0000", 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.98e-06 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010608+0000", 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4251e-05 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010617+0000", 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7819999999999999e-06 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010695+0000", 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8144000000000003e-05 2024-09-15T03:41:43.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010699+0000", 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8659999999999999e-06 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003696+0000", 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99299729199999998 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003710+0000", 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4001e-05 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.278305157999998, 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048967167, 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:43.295 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004683+0000", 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2899999999999999e-06 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004689+0000", 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9229999999999999e-06 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004703+0000", 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004719+0000", 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6095000000000001e-05 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005604+0000", 2024-09-15T03:41:43.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00088508100000000002 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005608+0000", 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.247e-06 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030447+0000", 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.024839515 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030545+0000", 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7453000000000004e-05 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.053649+0000", 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023104197999999999 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.300 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.278254566000001, 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.049664036, 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004733+0000", 2024-09-15T03:41:43.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0100000000000004e-07 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004737+0000", 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9049999999999999e-06 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004745+0000", 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7540000000000006e-06 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004752+0000", 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9000000000000006e-06 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005631+0000", 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087836100000000003 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005632+0000", 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.47e-07 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030868+0000", 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252366669999999 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030963+0000", 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4665000000000003e-05 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054396+0000", 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023433236 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.278228610999999, 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0501537240000001, 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.306 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004759+0000", 2024-09-15T03:41:43.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.82e-07 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004762+0000", 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2629999999999999e-06 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004769+0000", 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0260000000000002e-06 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004780+0000", 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0569e-05 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005657+0000", 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087720500000000004 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005658+0000", 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.159e-06 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031236+0000", 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0255778339999999 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031320+0000", 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4217999999999999e-05 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054912+0000", 2024-09-15T03:41:43.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023591867999999998 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:412 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~688128] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.199519092999999, 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013464116999999999, 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.312 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 412 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083474+0000", 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084200+0000", 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00072616300000000005 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084223+0000", 2024-09-15T03:41:43.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2501e-05 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084230+0000", 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8709999999999997e-06 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084286+0000", 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5943000000000003e-05 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084297+0000", 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1009000000000001e-05 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.089736+0000", 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054388459999999998 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.089847+0000", 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000110355 2024-09-15T03:41:43.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.090995+0000", 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011485 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094915+0000", 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039203249999999997 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.096868+0000", 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019529199999999999 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.096932+0000", 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3402e-05 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:413 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~540672] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:43.318 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.196345899000001, 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012499748, 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 413 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086643+0000", 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0509999999999999e-06 2024-09-15T03:41:43.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087150+0000", 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00050739100000000005 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087172+0000", 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1903e-05 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087182+0000", 2024-09-15T03:41:43.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087222+0000", 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0246000000000001e-05 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087227+0000", 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0019999999999997e-06 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.091754+0000", 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045271299999999999 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.091849+0000", 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4918000000000002e-05 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.092538+0000", 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068922000000000005 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095065+0000", 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025265919999999998 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099059+0000", 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039940599999999998 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099141+0000", 2024-09-15T03:41:43.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1749000000000006e-05 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:414 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~229376] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.195076643, 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01143303, 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.325 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 414 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087911+0000", 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2190000000000001e-06 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088148+0000", 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023696200000000001 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088161+0000", 2024-09-15T03:41:43.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3091000000000001e-05 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088167+0000", 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.48e-06 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088196+0000", 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9147999999999999e-05 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:43.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088200+0000", 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8060000000000001e-06 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093070+0000", 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048699090000000004 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093181+0000", 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000111192 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093854+0000", 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067265900000000004 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095066+0000", 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001212762 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099287+0000", 2024-09-15T03:41:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004220985 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099343+0000", 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5816999999999998e-05 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:415 3.19s0 3:9992a557:::smithi10536496-15:head [setxattr _header (57) in=7b] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.194811777000002, 2024-09-15T03:41:43.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012852831, 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 415 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9399999999999995e-07 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088178+0000", 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7439999999999999e-06 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088183+0000", 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1170000000000003e-06 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088186+0000", 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1109999999999999e-06 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088229+0000", 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2979999999999998e-05 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088237+0000", 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094281+0000", 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060443909999999997 2024-09-15T03:41:43.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094345+0000", 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3138000000000004e-05 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094761+0000", 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041665300000000001 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095343+0000", 2024-09-15T03:41:43.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058145699999999996 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.100956+0000", 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056129270000000002 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101028+0000", 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2167000000000002e-05 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:416 3.19s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.194796330999999, 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013816281999999999, 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 416 2024-09-15T03:41:43.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088191+0000", 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9900000000000002e-07 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088194+0000", 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5140000000000002e-06 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088201+0000", 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8070000000000002e-06 2024-09-15T03:41:43.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088204+0000", 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2389999999999998e-06 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088260+0000", 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5937999999999999e-05 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:43.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088266+0000", 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6330000000000001e-06 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094947+0000", 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066809970000000002 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095006+0000", 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8712000000000003e-05 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101269+0000", 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062632080000000001 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101315+0000", 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6714e-05 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.102007+0000", 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000691221 2024-09-15T03:41:43.343 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:423 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~704512] snapc 19=[19,18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.177724338999999, 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011973101, 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.344 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 423 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:43.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105264+0000", 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.928e-06 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106293+0000", 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010284280000000001 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106312+0000", 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8940000000000002e-05 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106320+0000", 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3610000000000001e-06 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106383+0000", 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:43.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106483+0000", 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9992000000000001e-05 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.107864+0000", 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013807120000000001 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109771+0000", 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019068659999999999 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.117186+0000", 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074148460000000001 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.117235+0000", 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9840999999999999e-05 2024-09-15T03:41:43.349 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:424 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~688128] snapc 19=[19,18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.109565+0000", 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.173421591, 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010381939, 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.350 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 424 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109565+0000", 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109565+0000", 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109567+0000", 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.889e-06 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110243+0000", 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067574099999999999 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110257+0000", 2024-09-15T03:41:43.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.401e-05 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110262+0000", 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7400000000000004e-06 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110298+0000", 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.608e-05 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110385+0000", 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6967999999999998e-05 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.112045+0000", 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016602500000000001 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.114536+0000", 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024912340000000002 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.119850+0000", 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053136900000000003 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.119947+0000", 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7337000000000007e-05 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.355 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:426 3.19s0 3:9992a557:::smithi10536496-15:head [setxattr _header (57) in=7b] snapc 19=[19,18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.110357+0000", 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.172630187999999, 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099373550000000001, 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 426 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.356 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110357+0000", 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110357+0000", 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110357+0000", 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110361+0000", 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110376+0000", 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5271999999999999e-05 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110381+0000", 2024-09-15T03:41:43.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6120000000000001e-06 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.114565+0000", 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041848390000000001 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.114682+0000", 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000116427 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.115269+0000", 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058693700000000005 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.116341+0000", 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001072508 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.120203+0000", 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038617679999999998 2024-09-15T03:41:43.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.120294+0000", 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0776e-05 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:427 3.19s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.110386+0000", 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.172600449000001, 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011215087, 2024-09-15T03:41:43.361 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 427 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110386+0000", 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110386+0000", 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110387+0000", 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0499999999999999e-07 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110390+0000", 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.022e-06 2024-09-15T03:41:43.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110395+0000", 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8949999999999997e-06 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110397+0000", 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6769999999999999e-06 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.115508+0000", 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051103499999999996 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.115605+0000", 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7818000000000004e-05 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.120670+0000", 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050643370000000004 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.120726+0000", 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6718999999999997e-05 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.121601+0000", 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087486400000000005 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.366 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:448 3.as0 3:5dde9339:::smithi10536496-16:head [rollback 17] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.243336412, 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012127656000000001, 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 448 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.367 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039651+0000", 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5000000000000003e-07 2024-09-15T03:41:43.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039654+0000", 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1209999999999998e-06 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039661+0000", 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4830000000000002e-06 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039664+0000", 2024-09-15T03:41:43.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.016e-06 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041239+0000", 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015745449999999999 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041330+0000", 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1223000000000004e-05 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041870+0000", 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00053994899999999996 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043742+0000", 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018719940000000001 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.051733+0000", 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079906860000000003 2024-09-15T03:41:43.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.051778+0000", 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5089000000000001e-05 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:449 3.as0 3:5dde9339:::smithi10536496-16:head [append 0~0] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.243318715999999, 2024-09-15T03:41:43.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018300219999999999, 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 449 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.89e-07 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039671+0000", 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2910000000000002e-06 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039676+0000", 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.383e-06 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039680+0000", 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4879999999999999e-06 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042268+0000", 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025880899999999999 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042414+0000", 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000146312 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042875+0000", 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00046076800000000002 2024-09-15T03:41:43.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044451+0000", 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001575833 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.057930+0000", 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013479326 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.057968+0000", 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.824e-05 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/51 ECSubWriteReply(tid=49, last_complete=65'44, committed=1, applied=1))", 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.241310125, 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.47081936099999999, 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.378 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0900000000000001e-07 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041681+0000", 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8970000000000001e-06 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512389+0000", 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.470707388 2024-09-15T03:41:43.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512401+0000", 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2897999999999999e-05 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512468+0000", 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6607999999999997e-05 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512496+0000", 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7861e-05 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:625 3.as0 3:57d09725:::smithi10536496-48:head [append 0~475136] snapc 28=[28,27,26,25,24] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.25456549699999997, 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011332543, 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.382 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 625 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028422+0000", 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2160000000000001e-06 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028856+0000", 2024-09-15T03:41:43.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043326999999999998 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028866+0000", 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028871+0000", 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028939+0000", 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8388999999999999e-05 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029004+0000", 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4908999999999998e-05 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029559+0000", 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000555465 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030619+0000", 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010596799999999999 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039713+0000", 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090945100000000001 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039754+0000", 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0303999999999999e-05 2024-09-15T03:41:43.387 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:626 3.as0 3:57d09725:::smithi10536496-48:head [setxattr _header (57) in=7b] snapc 28=[28,27,26,25,24] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.25410803300000001, 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011104786, 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.388 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 626 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028881+0000", 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.069e-06 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028886+0000", 2024-09-15T03:41:43.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028890+0000", 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.963e-06 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029913+0000", 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010229080000000001 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030002+0000", 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8792e-05 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030403+0000", 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00040096199999999999 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030723+0000", 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00032022399999999998 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039938+0000", 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0092151850000000007 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039983+0000", 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5018000000000002e-05 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.393 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:627 3.as0 3:57d09725:::smithi10536496-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.25408153300000003, 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012208765, 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 627 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028906+0000", 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028907+0000", 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.544e-06 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028911+0000", 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028914+0000", 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:41:43.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030561+0000", 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016473379999999999 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030614+0000", 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2911000000000002e-05 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.040187+0000", 2024-09-15T03:41:43.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095729660000000005 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.040275+0000", 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7498999999999997e-05 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.041114+0000", 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000839151 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.398 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.399 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.399 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.399 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:43.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:43.401 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:43.399+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:43.500 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:43.499+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:43.543 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:43.544 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:43.544 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:43.544 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:43.552 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:43.601 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:43.600+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:43.694 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:43.695 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:43.695 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:43.695 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:43.695 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:43.702 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:43.701+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:43.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:43.803 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:43.802+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.786189399000001, 2024-09-15T03:41:43.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023454370000000001, 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:43.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018416+0000", 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5300000000000004e-07 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018457+0000", 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021725800000000001 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018754+0000", 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0282000000000001e-05 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018806+0000", 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018858+0000", 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1873000000000003e-05 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020727+0000", 2024-09-15T03:41:43.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001868897 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020761+0000", 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4165999999999998e-05 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=1, reqid=client.4562.0:8, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.784248536, 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021286080000000001, 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020356+0000", 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020357+0000", 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0139999999999999e-06 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020398+0000", 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1109000000000002e-05 2024-09-15T03:41:43.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020456+0000", 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7636999999999997e-05 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020465+0000", 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6230000000000003e-06 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020522+0000", 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7314e-05 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020532+0000", 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0200999999999999e-05 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022453+0000", 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019207950000000001 2024-09-15T03:41:43.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.022485+0000", 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1915000000000002e-05 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:43.816 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.776464021999999, 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020128619999999998, 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:43.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028141+0000", 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.49e-07 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028149+0000", 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4889999999999995e-06 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028822+0000", 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067247100000000001 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028830+0000", 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1429999999999998e-06 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0082999999999998e-05 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.028930+0000", 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0395e-05 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030125+0000", 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011949440000000001 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030154+0000", 2024-09-15T03:41:43.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8187999999999998e-05 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.733660651999998, 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084043045999999996, 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070945+0000", 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.124e-06 2024-09-15T03:41:43.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070959+0000", 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4212999999999999e-05 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154508+0000", 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083548836000000001 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154515+0000", 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154566+0000", 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1025e-05 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154578+0000", 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-09-15T03:41:43.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154965+0000", 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038690300000000002 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154987+0000", 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2172999999999999e-05 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=24, reqid=client.4562.0:100, at_version=35'3, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.721130561999999, 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002110823, 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083474+0000", 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083476+0000", 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.807e-06 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083643+0000", 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016675300000000001 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083675+0000", 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1962000000000003e-05 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083684+0000", 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6930000000000002e-06 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083746+0000", 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2008000000000007e-05 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083756+0000", 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9450000000000005e-06 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085554+0000", 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017979980000000001 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085585+0000", 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0657000000000001e-05 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=12, reqid=client.4562.0:150, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:43.830 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.650360393, 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035676890000000002, 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154245+0000", 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154487+0000", 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000242308 2024-09-15T03:41:43.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156728+0000", 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022401589999999998 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156736+0000", 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3259999999999996e-06 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156781+0000", 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4583000000000003e-05 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156789+0000", 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8359999999999998e-06 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157761+0000", 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097187399999999996 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157812+0000", 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0695999999999997e-05 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=14, reqid=client.4562.0:152, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.647416503000002, 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028494564, 2024-09-15T03:41:43.835 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157189+0000", 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3399999999999998e-07 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157206+0000", 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7207e-05 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185008+0000", 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027801691999999999 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185015+0000", 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185068+0000", 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2333000000000002e-05 2024-09-15T03:41:43.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185077+0000", 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0259999999999996e-06 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185660+0000", 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058343299999999996 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185683+0000", 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714e-05 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=15, reqid=client.4562.0:164, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.619936733999999, 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027286849999999998, 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.840 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184669+0000", 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000005e-07 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184986+0000", 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000317027 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015847960000000001 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186578+0000", 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3220000000000004e-06 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186619+0000", 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1186000000000002e-05 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186629+0000", 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187377+0000", 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074841899999999995 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187397+0000", 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9233999999999999e-05 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=17, reqid=client.4562.0:166, at_version=35'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.617719751999999, 2024-09-15T03:41:43.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099130431000000005, 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:43.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186886+0000", 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186906+0000", 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0482000000000001e-05 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285393+0000", 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098487125999999994 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285401+0000", 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3780000000000003e-06 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285446+0000", 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028999999999999e-05 2024-09-15T03:41:43.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285453+0000", 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285995+0000", 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00054192900000000002 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.286015+0000", 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0579000000000001e-05 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 32.783091567, 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030110250000000001, 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.851 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8299999999999998e-07 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022295+0000", 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078131100000000003 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022331+0000", 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6103000000000003e-05 2024-09-15T03:41:43.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022338+0000", 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6410000000000002e-06 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022426+0000", 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7366000000000003e-05 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022437+0000", 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.131e-05 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024502+0000", 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002065271 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024524+0000", 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1739999999999999e-05 2024-09-15T03:41:43.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 47/32 ECSubWrite(tid=66, reqid=osd.11.0:66, at_version=47'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.796073644, 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003545678, 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:43.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.29e-07 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008562+0000", 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0871999999999999e-05 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008606+0000", 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4057999999999997e-05 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008618+0000", 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878999999999999e-05 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008722+0000", 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104236 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008732+0000", 2024-09-15T03:41:43.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0142999999999999e-05 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012049+0000", 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033168920000000001 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012077+0000", 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7368999999999998e-05 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.860 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 50/32 ECSubWrite(tid=74, reqid=client.4562.0:320, at_version=50'21, trim_to=0'0, roll_forward_to=47'18))", 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:17.084717+0000", 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.719887916000001, 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021220940000000002, 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084717+0000", 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084717+0000", 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.084718+0000", 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5000000000000001e-07 2024-09-15T03:41:43.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085070+0000", 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00035266600000000001 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085107+0000", 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6232000000000001e-05 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085116+0000", 2024-09-15T03:41:43.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6379999999999999e-06 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085165+0000", 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8507000000000002e-05 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.085174+0000", 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7869999999999996e-06 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086815+0000", 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001640861 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.086839+0000", 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3553e-05 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.780064510999999, 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042098190000000001, 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9600000000000001e-07 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024564+0000", 2024-09-15T03:41:43.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3714e-05 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024603+0000", 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9048000000000003e-05 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024610+0000", 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1520000000000003e-06 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024717+0000", 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000106872 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024728+0000", 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028684+0000", 2024-09-15T03:41:43.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039562190000000004 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028750+0000", 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5658000000000001e-05 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.18s6 51/49 ECSubWrite(tid=153, reqid=client.4562.0:369, at_version=51'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:43.870 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.769374314, 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028221240000000001, 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:43.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035231+0000", 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4699999999999998e-07 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036344+0000", 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001113721 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036395+0000", 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0247000000000002e-05 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036403+0000", 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4630000000000001e-06 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036467+0000", 2024-09-15T03:41:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3876999999999998e-05 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036480+0000", 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2818e-05 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.038029+0000", 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015492629999999999 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.038052+0000", 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3487999999999999e-05 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 56/32 ECSubWrite(tid=88, reqid=osd.11.0:88, at_version=56'26, trim_to=0'0, roll_forward_to=51'24))", 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 20.741386262999999, 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003747588, 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.875 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063219+0000", 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063262+0000", 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3173000000000002e-05 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063457+0000", 2024-09-15T03:41:43.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019513700000000001 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063464+0000", 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6120000000000004e-06 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063517+0000", 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3477999999999999e-05 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063528+0000", 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.112e-05 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066940+0000", 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034120299999999999 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066966+0000", 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5724e-05 2024-09-15T03:41:43.879 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 61/32 ECSubWrite(tid=91, reqid=client.4562.0:431, at_version=61'16, trim_to=0'0, roll_forward_to=52'14))", 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.111024+0000", 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.693581014999999, 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019098050000000001, 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.880 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111024+0000", 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111024+0000", 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111024+0000", 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1300000000000001e-07 2024-09-15T03:41:43.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111054+0000", 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9638e-05 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111130+0000", 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6516999999999999e-05 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111140+0000", 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8970000000000003e-06 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111210+0000", 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9833999999999996e-05 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.111221+0000", 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1314e-05 2024-09-15T03:41:43.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.112900+0000", 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016783830000000001 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.112933+0000", 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3908999999999997e-05 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 62/32 ECSubWrite(tid=60, reqid=osd.2.0:60, at_version=62'32, trim_to=0'0, roll_forward_to=58'30))", 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.762414063, 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025058089999999999, 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.886 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042217+0000", 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5256e-05 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042269+0000", 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2383999999999999e-05 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042281+0000", 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1977000000000001e-05 2024-09-15T03:41:43.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042333+0000", 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2537999999999997e-05 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042345+0000", 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1307e-05 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.888 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044670+0000", 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023257120000000002 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044696+0000", 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5996e-05 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 62/32 ECSubWrite(tid=95, reqid=client.4562.0:446, at_version=62'18, trim_to=0'0, roll_forward_to=61'16))", 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.043111+0000", 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 14.761494142, 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019235120000000001, 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043111+0000", 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043111+0000", 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043111+0000", 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043144+0000", 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2741999999999999e-05 2024-09-15T03:41:43.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043193+0000", 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8896000000000001e-05 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043200+0000", 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8530000000000003e-06 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.892 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044013+0000", 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00081332500000000005 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044022+0000", 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7770000000000008e-06 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.045006+0000", 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00098357400000000008 2024-09-15T03:41:43.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.045034+0000", 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8469000000000001e-05 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.7489332280000003, 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025217009999999999, 2024-09-15T03:41:43.894 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055672+0000", 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.18e-07 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056050+0000", 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00037845499999999998 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.896 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056164+0000", 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113744 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056168+0000", 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056220+0000", 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2176999999999999e-05 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.897 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056230+0000", 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8500000000000006e-06 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058166+0000", 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001936243 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058193+0000", 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7137000000000001e-05 2024-09-15T03:41:43.898 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4175.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.056089+0000", 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 9.7485160680000007, 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020217830000000001, 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4175", 2024-09-15T03:41:43.899 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/1675871746", 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 1 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056089+0000", 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056089+0000", 2024-09-15T03:41:43.900 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056090+0000", 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8610000000000001e-06 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056099+0000", 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2570000000000007e-06 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.901 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056115+0000", 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6483000000000001e-05 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056128+0000", 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2629999999999999e-05 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056190+0000", 2024-09-15T03:41:43.902 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2311999999999996e-05 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056198+0000", 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6879999999999999e-06 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058017+0000", 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018188379999999999 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:43.903 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058110+0000", 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3714000000000004e-05 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:43.904 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:43.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:43.906 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:43.904+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:44.006 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:44.005+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:44.022 INFO:tasks.rados.rados.0.smithi105.stdout:263: done (3 left) 2024-09-15T03:41:44.022 INFO:tasks.rados.rados.0.smithi105.stdout:264: copy_from oid 20 from oid 38 current snap is 22 2024-09-15T03:41:44.022 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 43 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-15T03:41:44.022 INFO:tasks.rados.rados.0.smithi105.stdout:260: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-15T03:41:44.024 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:44.024 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:44.035 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:41:44.036 INFO:tasks.rados.rados.0.smithi105.stdout:262: finishing write tid 1 to smithi10536496-48 2024-09-15T03:41:44.036 INFO:tasks.rados.rados.0.smithi105.stdout:262: finishing write tid 2 to smithi10536496-48 2024-09-15T03:41:44.036 INFO:tasks.rados.rados.0.smithi105.stdout:262: finishing write tid 3 to smithi10536496-48 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:260: done (3 left) 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:261: done (2 left) 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:265: copy_from oid 36 from oid 39 current snap is 22 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:262: finishing write tid 4 to smithi10536496-48 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:262: finishing write tid 5 to smithi10536496-48 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 57 (ObjNum 114 snap 22 seq_num 114) dirty exists 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:262: left oid 48 (ObjNum 114 snap 22 seq_num 114) 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:264: finishing copy_from to smithi10536496-20 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:264: got expected ENOENT (src dne) 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:262: done (2 left) 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:266: read oid 42 snap 21 2024-09-15T03:41:44.037 INFO:tasks.rados.rados.0.smithi105.stdout:266: expect (ObjNum 106 snap 17 seq_num 106) 2024-09-15T03:41:44.038 INFO:tasks.rados.rados.0.smithi105.stdout:267: rmattr oid 10 current snap is 22 2024-09-15T03:41:44.038 INFO:tasks.rados.rados.0.smithi105.stdout:267: done (3 left) 2024-09-15T03:41:44.038 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 29 current snap is 22 2024-09-15T03:41:44.038 INFO:tasks.rados.rados.0.smithi105.stdout:268: seq_num 115 ranges {98304=770048,868352=704512,1572864=458752} 2024-09-15T03:41:44.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:44.042 INFO:tasks.rados.rados.0.smithi105.stdout:268: writing smithi10536496-29 from 98304 to 868352 tid 1 2024-09-15T03:41:44.046 INFO:tasks.rados.rados.0.smithi105.stdout:268: writing smithi10536496-29 from 868352 to 1572864 tid 2 2024-09-15T03:41:44.049 INFO:tasks.rados.rados.0.smithi105.stdout:268: writing smithi10536496-29 from 1572864 to 2031616 tid 3 2024-09-15T03:41:44.049 INFO:tasks.rados.rados.0.smithi105.stdout:269: snap_remove snap 19 2024-09-15T03:41:44.106 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:44.105+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:44.146 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:44.304 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:44.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:44.307 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:44.304+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:44.407 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:44.406+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.392372231000003, 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094904819999999997, 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.23e-07 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017537+0000", 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8838000000000003e-05 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017912+0000", 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037478400000000002 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017919+0000", 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8010000000000004e-06 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017987+0000", 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7138000000000007e-05 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017998+0000", 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0908e-05 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026958+0000", 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089603229999999992 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026988+0000", 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0467000000000002e-05 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:5, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.391646172999998, 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085434570000000008, 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018225+0000", 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2619999999999999e-06 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018254+0000", 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8656000000000001e-05 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018297+0000", 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999999e-05 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018310+0000", 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3052e-05 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018392+0000", 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1527999999999996e-05 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4590000000000004e-06 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026715+0000", 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083134909999999992 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026767+0000", 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599000000000002e-05 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.391101902999999, 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080389840000000008, 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018768+0000", 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1199999999999999e-07 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018811+0000", 2024-09-15T03:41:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2938000000000001e-05 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018916+0000", 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010431 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018923+0000", 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019065+0000", 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014137500000000001 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019075+0000", 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0271e-05 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026774+0000", 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076987230000000002 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026807+0000", 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3655999999999998e-05 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.388261798999999, 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010778574000000001, 2024-09-15T03:41:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021609+0000", 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021692+0000", 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999997e-05 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021735+0000", 2024-09-15T03:41:44.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3581000000000002e-05 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021743+0000", 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3119999999999994e-06 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021822+0000", 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8437999999999997e-05 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021830+0000", 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2989999999999997e-06 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032339+0000", 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010508768999999999 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032387+0000", 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7851e-05 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.387609015000002, 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010171065999999999, 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:44.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022262+0000", 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.043e-06 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022318+0000", 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6208000000000001e-05 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022353+0000", 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4437000000000002e-05 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022362+0000", 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:41:44.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022408+0000", 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5775000000000001e-05 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022419+0000", 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1450999999999999e-05 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032351+0000", 2024-09-15T03:41:44.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099319260000000006 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032432+0000", 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1032000000000006e-05 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:44.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.386941514999997, 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094003279999999995, 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022930+0000", 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022949+0000", 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9916e-05 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022995+0000", 2024-09-15T03:41:44.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5173000000000003e-05 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023004+0000", 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2990000000000002e-06 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023067+0000", 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023073+0000", 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032300+0000", 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092268320000000008 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032329+0000", 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8779999999999999e-05 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.386205809000003, 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087825990000000003, 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:44.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023665+0000", 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5199999999999995e-07 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023681+0000", 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-05 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023713+0000", 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1726000000000001e-05 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023720+0000", 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6830000000000002e-06 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023765+0000", 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4988999999999999e-05 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023771+0000", 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9109999999999998e-06 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032437+0000", 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086659760000000006 2024-09-15T03:41:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032447+0000", 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0222e-05 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.386156036999999, 2024-09-15T03:41:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087050259999999994, 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023715+0000", 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023733+0000", 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8442999999999999e-05 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023770+0000", 2024-09-15T03:41:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6585999999999998e-05 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023784+0000", 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4440999999999999e-05 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023829+0000", 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5099000000000003e-05 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023836+0000", 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4400000000000002e-06 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032396+0000", 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085599330000000005 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032419+0000", 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3288000000000001e-05 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.385576829999998, 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081451809999999996, 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024294+0000", 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024371+0000", 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6451000000000004e-05 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024415+0000", 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4292000000000002e-05 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024423+0000", 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0700000000000007e-06 2024-09-15T03:41:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024479+0000", 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024488+0000", 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1109999999999997e-06 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032423+0000", 2024-09-15T03:41:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079349969999999992 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032438+0000", 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5279000000000001e-05 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.382729466000001, 2024-09-15T03:41:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013181325000000001, 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7000000000000004e-07 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027155+0000", 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027190+0000", 2024-09-15T03:41:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5738000000000002e-05 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027199+0000", 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1999999999999994e-06 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027257+0000", 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8690000000000002e-05 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027265+0000", 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8860000000000004e-06 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040256+0000", 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012990323999999999 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040322+0000", 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6392999999999999e-05 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.382250761999998, 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012779128000000001, 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027620+0000", 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.32e-07 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027692+0000", 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000006e-05 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027773+0000", 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0605000000000005e-05 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027782+0000", 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027884+0000", 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010173100000000001 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027894+0000", 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9210000000000004e-06 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040357+0000", 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463284999999999 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1026000000000003e-05 2024-09-15T03:41:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.380963057999999, 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011382537, 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028908+0000", 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0300000000000001e-06 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028922+0000", 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4329e-05 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028975+0000", 2024-09-15T03:41:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2573e-05 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028983+0000", 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029027+0000", 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3912000000000002e-05 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029036+0000", 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9560000000000003e-06 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040254+0000", 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011217632999999999 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5880999999999997e-05 2024-09-15T03:41:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.380950325000001, 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011534467, 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028921+0000", 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.057e-06 2024-09-15T03:41:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028959+0000", 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8217999999999999e-05 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028993+0000", 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4425999999999999e-05 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029011+0000", 2024-09-15T03:41:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7465e-05 2024-09-15T03:41:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029270+0000", 2024-09-15T03:41:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025923899999999998 2024-09-15T03:41:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029280+0000", 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040431+0000", 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011151283 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040454+0000", 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3309e-05 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.380312263999997, 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013707001, 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029559+0000", 2024-09-15T03:41:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029676+0000", 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117273 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029720+0000", 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4026000000000002e-05 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029728+0000", 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3240000000000003e-06 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030022+0000", 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029338200000000002 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030037+0000", 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5082000000000001e-05 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043234+0000", 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013196548000000001 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043265+0000", 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1312000000000001e-05 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.379251824000001, 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097244270000000008, 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030619+0000", 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999998e-07 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030630+0000", 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030662+0000", 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1813000000000002e-05 2024-09-15T03:41:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030680+0000", 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8306999999999999e-05 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030722+0000", 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1823000000000003e-05 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030729+0000", 2024-09-15T03:41:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040325+0000", 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095952650000000004 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040343+0000", 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8079000000000001e-05 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.378260322000003, 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011724098000000001, 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031742+0000", 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013129600000000001 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031779+0000", 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7311000000000002e-05 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031787+0000", 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031842+0000", 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4617999999999998e-05 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031856+0000", 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.42e-05 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043313+0000", 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011457292000000001 2024-09-15T03:41:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043334+0000", 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=9, reqid=client.4562.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.378061879999997, 2024-09-15T03:41:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011489871, 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031809+0000", 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0130000000000001e-06 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031830+0000", 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0801999999999998e-05 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031864+0000", 2024-09-15T03:41:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3575000000000003e-05 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031869+0000", 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5029999999999997e-06 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032118+0000", 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024897299999999999 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032128+0000", 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4879999999999996e-06 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043276+0000", 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01114878 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043298+0000", 2024-09-15T03:41:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1736999999999999e-05 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.377391267, 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010878669000000001, 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:41:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032507+0000", 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7140000000000001e-05 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032542+0000", 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4986e-05 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032549+0000", 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2130000000000002e-06 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034852+0000", 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023031089999999998 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034862+0000", 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0156000000000001e-05 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043343+0000", 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084812489999999997 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043357+0000", 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4219000000000001e-05 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.374749235000003, 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008115901, 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035122+0000", 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035140+0000", 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7915999999999999e-05 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035176+0000", 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6335999999999999e-05 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035184+0000", 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035238+0000", 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4048000000000002e-05 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035248+0000", 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043207+0000", 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079593690000000009 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043237+0000", 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9788e-05 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=20, reqid=client.4562.0:159, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.249451628000003, 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091519410000000002, 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160419+0000", 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7499999999999999e-07 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160707+0000", 2024-09-15T03:41:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028803500000000002 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160722+0000", 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5235000000000001e-05 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160727+0000", 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7600000000000002e-06 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160775+0000", 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7732999999999999e-05 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160785+0000", 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0380999999999999e-05 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169551+0000", 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087657610000000004 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169570+0000", 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9261000000000001e-05 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.507 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:44.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:44.509 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:44.508+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:44.609 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:44.608+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:44.610 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:44.610 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:44.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:44.710 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:44.709+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:44.720 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:44.720 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:44.720 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:44.721 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:44.721 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:44.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:44.811 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:44.810+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:2 3.8s0 3:136df151:::smithi10536496-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.813975630999998, 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012099814, 2024-09-15T03:41:44.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.99e-07 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016626+0000", 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:41:44.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016636+0000", 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8439999999999998e-06 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018474+0000", 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018381859999999999 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018568+0000", 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4424000000000003e-05 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019035+0000", 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046713100000000002 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024240+0000", 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052042240000000003 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028633+0000", 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043929340000000002 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028722+0000", 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9912000000000002e-05 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:3 3.8s0 3:136df151:::smithi10536496-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.813959777999997, 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016082863999999999, 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016639+0000", 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016640+0000", 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2729999999999999e-06 2024-09-15T03:41:44.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.113e-06 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016650+0000", 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3489999999999999e-06 2024-09-15T03:41:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019489+0000", 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028386459999999998 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019567+0000", 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7417999999999996e-05 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029120+0000", 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095539030000000007 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029205+0000", 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4301000000000005e-05 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032721+0000", 2024-09-15T03:41:44.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003516576 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:63 3.8s0 3:12b3a6c7:::smithi10536496-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.786865644000002, 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012980953, 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 63 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.72e-07 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8279999999999999e-06 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043740+0000", 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9110000000000001e-06 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043742+0000", 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6460000000000002e-06 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045685+0000", 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001942706 2024-09-15T03:41:44.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045767+0000", 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2405999999999997e-05 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054190+0000", 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084225670000000006 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054302+0000", 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112184 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.056714+0000", 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024113329999999999 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.850 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:108 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.739781037999997, 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010882141999999999, 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090817+0000", 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090818+0000", 2024-09-15T03:41:44.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.39e-07 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090819+0000", 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8080000000000001e-06 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090824+0000", 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7419999999999997e-06 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.090828+0000", 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7890000000000001e-06 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.093335+0000", 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025070370000000002 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.093382+0000", 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7268999999999998e-05 2024-09-15T03:41:44.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096751+0000", 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033692040000000002 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0794999999999998e-05 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.101699+0000", 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049071590000000003 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:286 3.8s0 3:136df151:::smithi10536496-1:head [create,setxattr dj[s][nsgcdlaO`ntnnogS`sZdZO[V\\]llWi\\gjj (11296) in=40b,setxattr qRNaVOjbUaQsk\\XsiZbOhrsOSoqX_Tqndtkb\\_Sa (1297) in=40b,setxattr `hdQPrsh[aUo`rPcbh\\tWT\\PatkS^p[P_Pdmcgdi (840) in=40b,setxattr RXicf\\^UcqO_mXh^am]mfcdttd\\mfOiPqZcomOo` (1822) in=40b,setxattr oZlXqWbdtOrc_U_m[apaV_Nr[hOOUgj`sVNpq^Rp (1473) in=40b,setxattr TrdTbcloOaiWXhjih\\mdXfYZcg\\^aOrnVOj^sf`s (10104) in=40b,setxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt (1263) in=40b,setxattr ^qaVseQnk[`qOXVa`m]]NtQ\\psXaNYand^tQYrWt (1051) in=40b,setxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q (418) in=40b,setxattr cfoPTk\\o\\Qqbi^XNOW[dbpcRb`OledY]YqcmmqUr (19016) in=40b,setxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd (10104) in=35b,setxattr mtVO^NRpeT]rQZYdTadneUO_^VXn^UdhZ^\\bUQiR (769) in=40b,setxattr prahkiNnm`U[rXSatop\\lk_p\\WVkdatg[XURZrsm (4560) in=40b,setxattr isZNUepUtPtRnYQaXpfelSVVR\\apSohRfqcfV`T_ (10033) in=40b,setxattr QZkproN[Zssrnra[rrbkaSXpatcreQpt`tWbOdoO (8349) in=40b,setxattr YqVPkaeiY^Tbe\\Rr\\heXkWjtOsi`iaN[TO\\^kP^] (876) in=40b,setxattr ZdfcZ]]snWVj]dZhZ]YSjVneRRoNbk`d^een^nqN (3368) in=40b,setxattr Y^XWcNfadqaYlbNpPbVRoPtgWreNtPrhWlfbbYY] (10804) in=40b,setxattr a`l_j\\^WlXY[ekSUURUhO (10104) in=21b,setxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ (1541) in=40b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.802364877999999, 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025921639, 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 286 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028235+0000", 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.872e-06 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028332+0000", 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-09-15T03:41:44.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028352+0000", 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0262999999999999e-05 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028359+0000", 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4390000000000003e-06 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029687+0000", 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013271909999999999 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029881+0000", 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000194792 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.031152+0000", 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012703910000000001 2024-09-15T03:41:44.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.033779+0000", 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002626724 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054105+0000", 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020326455 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054155+0000", 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0021000000000001e-05 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=0, reqid=client.4562.0:300, at_version=50'10, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.795323241999998, 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012435007, 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.862 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6899999999999999e-07 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037696+0000", 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024206610000000002 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038322+0000", 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062559400000000004 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038330+0000", 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8789999999999992e-06 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038389+0000", 2024-09-15T03:41:44.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8168999999999998e-05 2024-09-15T03:41:44.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038400+0000", 2024-09-15T03:41:44.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1083999999999999e-05 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.047676+0000", 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092765030000000002 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.047710+0000", 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3847999999999998e-05 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=1, reqid=client.4562.0:301, at_version=50'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.792897481000001, 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011111194, 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:44.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0800000000000004e-07 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037845+0000", 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143296 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038740+0000", 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089520099999999998 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038747+0000", 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1459999999999999e-06 2024-09-15T03:41:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.039546+0000", 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000799082 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.039557+0000", 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048784+0000", 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092276149999999998 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048812+0000", 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7477000000000001e-05 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=2, reqid=client.4562.0:302, at_version=50'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.79275002, 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01098676, 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037849+0000", 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-07 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037903+0000", 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4562999999999999e-05 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038780+0000", 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087622000000000004 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038785+0000", 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1189999999999996e-06 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.044855+0000", 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060705159999999998 2024-09-15T03:41:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.044866+0000", 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0355e-05 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048821+0000", 2024-09-15T03:41:44.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039556799999999996 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048835+0000", 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3805000000000001e-05 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=3, reqid=client.4562.0:303, at_version=50'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:44.879 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.792691689000002, 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011244417, 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037921+0000", 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3678e-05 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038795+0000", 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000874331 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038799+0000", 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3930000000000001e-06 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045279+0000", 2024-09-15T03:41:44.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064799879999999999 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045289+0000", 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3510000000000008e-06 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048867+0000", 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003578178 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.049151+0000", 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028403800000000002 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:357 3.8s0 3:115f2b89:::smithi10536496-36:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.668698953, 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012091859999999999, 2024-09-15T03:41:44.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 357 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161901+0000", 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5510000000000001e-06 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161907+0000", 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161921+0000", 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.393e-05 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161926+0000", 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1309999999999997e-06 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161969+0000", 2024-09-15T03:41:44.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2123000000000003e-05 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.162082+0000", 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011328999999999999 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.170186+0000", 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081040480000000008 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.171217+0000", 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010315389999999999 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.173948+0000", 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027302070000000001 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.173991+0000", 2024-09-15T03:41:44.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3572000000000002e-05 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.805670941999999, 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011226435999999999, 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024928+0000", 2024-09-15T03:41:44.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3099999999999996e-07 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024961+0000", 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2818000000000003e-05 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025047+0000", 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6197000000000002e-05 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025054+0000", 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3029999999999996e-06 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025108+0000", 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4744000000000003e-05 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025120+0000", 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1212999999999999e-05 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.036125+0000", 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011005910000000001 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.036154+0000", 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.832e-05 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 51/49 ECSubWrite(tid=10, reqid=osd.10.0:10, at_version=51'19, trim_to=0'0, roll_forward_to=50'17))", 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.805292578, 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013366444, 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:44.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025307+0000", 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025333+0000", 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6886999999999999e-05 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025376+0000", 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2821000000000002e-05 2024-09-15T03:41:44.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025384+0000", 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025562+0000", 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017756799999999999 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025571+0000", 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-06 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038645+0000", 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013073817999999999 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038672+0000", 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7131000000000002e-05 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:438 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~753664] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.796890785, 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025886468999999999, 2024-09-15T03:41:44.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 438 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033710+0000", 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4380000000000002e-06 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034822+0000", 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011121359999999999 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034849+0000", 2024-09-15T03:41:44.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7406000000000002e-05 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034859+0000", 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6830000000000003e-06 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034943+0000", 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4303999999999998e-05 2024-09-15T03:41:44.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034953+0000", 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5019999999999998e-06 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039409+0000", 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044563520000000002 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039523+0000", 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113514 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.041482+0000", 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019589410000000001 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048318+0000", 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006836033 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059547+0000", 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011228802 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059594+0000", 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7358000000000003e-05 2024-09-15T03:41:44.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:439 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~720896] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.790619241, 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019746224, 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 439 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039981+0000", 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829999999999999e-06 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042101+0000", 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021201850000000001 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042116+0000", 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4994000000000001e-05 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042124+0000", 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7729999999999999e-06 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043391+0000", 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012674959999999999 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043512+0000", 2024-09-15T03:41:44.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120469 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.044649+0000", 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001137327 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048320+0000", 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036706320000000001 2024-09-15T03:41:44.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059680+0000", 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011360093 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059725+0000", 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5472000000000001e-05 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:440 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~49152] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.78842818, 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02155733, 2024-09-15T03:41:44.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 440 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:44.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042171+0000", 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.39e-06 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042243+0000", 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1474e-05 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042256+0000", 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3016999999999999e-05 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042262+0000", 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4269999999999996e-06 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045435+0000", 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031722920000000002 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045540+0000", 2024-09-15T03:41:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105648 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046184+0000", 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064343199999999999 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049337+0000", 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003153089 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063687+0000", 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01435036 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063727+0000", 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0200999999999997e-05 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:441 3.8s0 3:115f2b89:::smithi10536496-36:head [setxattr _header (57) in=7b] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.788327748, 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021588698, 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 441 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5499999999999997e-07 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042275+0000", 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7900000000000001e-06 2024-09-15T03:41:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042284+0000", 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8440000000000004e-06 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042294+0000", 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6569999999999992e-06 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046742+0000", 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044484989999999999 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046840+0000", 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7651000000000004e-05 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047278+0000", 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043781499999999998 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049368+0000", 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020900620000000002 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063805+0000", 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014437324 2024-09-15T03:41:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063859+0000", 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4301000000000001e-05 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:442 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.788299384, 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028656429000000001, 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 442 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042302+0000", 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4990000000000001e-06 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042309+0000", 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7139999999999996e-06 2024-09-15T03:41:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042313+0000", 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2660000000000003e-06 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047492+0000", 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051795030000000002 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047573+0000", 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0895000000000004e-05 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064140+0000", 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016566658000000001 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064227+0000", 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199000000000003e-05 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.070955+0000", 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067281440000000001 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:559 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~3489792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:38.128514+0000", 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.7020843379999997, 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0109283, 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 559 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.128514+0000", 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.128514+0000", 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.128514+0000", 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3399999999999999e-07 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.128518+0000", 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.856e-06 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.128527+0000", 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6589999999999996e-06 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.128532+0000", 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5900000000000001e-06 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.129371+0000", 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083936600000000003 2024-09-15T03:41:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.129448+0000", 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6971999999999994e-05 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:38.139442+0000", 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099943229999999994 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:631 3.8s0 3:115f2b89:::smithi10536496-36:head [copy-from ver 41] snapc 28=[28,27,26,25] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.79441799599999996, 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.46301878800000001, 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 631 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036182+0000", 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9750000000000001e-06 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036188+0000", 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8980000000000001e-06 2024-09-15T03:41:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036204+0000", 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5906e-05 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036211+0000", 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6490000000000002e-06 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036287+0000", 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5835999999999997e-05 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036393+0000", 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106378 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.057720+0000", 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021326686000000001 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.060505+0000", 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027857680000000001 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.499151+0000", 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.43864593699999999 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.499199+0000", 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7754999999999999e-05 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:632 3.8s0 3:115f2b89:::smithi10536496-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.79437192400000001, 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023198263, 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 632 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036227+0000", 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7199999999999998e-07 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036230+0000", 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7949999999999999e-06 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036238+0000", 2024-09-15T03:41:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1829999999999994e-06 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036242+0000", 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740000000000004e-06 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036579+0000", 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033680299999999999 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036635+0000", 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6545999999999997e-05 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059280+0000", 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022644272 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059341+0000", 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1425000000000002e-05 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059425+0000", 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3592999999999997e-05 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:44.957 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:44.958 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:44.957+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:44.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:44.975 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [3] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15, 3] 2024-09-15T03:41:44.976 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:41:44.976 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-15T03:41:44.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-09-15T03:41:45.020 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:41:45.020 INFO:tasks.rados.rados.0.smithi105.stdout:269: done (4 left) 2024-09-15T03:41:45.021 INFO:tasks.rados.rados.0.smithi105.stdout:270: rmattr oid 37 current snap is 22 2024-09-15T03:41:45.021 INFO:tasks.rados.rados.0.smithi105.stdout:270: done (4 left) 2024-09-15T03:41:45.021 INFO:tasks.rados.rados.0.smithi105.stdout:271: rmattr oid 4 current snap is 22 2024-09-15T03:41:45.021 INFO:tasks.rados.rados.0.smithi105.stdout:264: finishing copy_from racing read to smithi10536496-20 2024-09-15T03:41:45.021 INFO:tasks.rados.rados.0.smithi105.stdout:266: expect (ObjNum 106 snap 17 seq_num 106) 2024-09-15T03:41:45.059 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:45.058+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:45.111 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:45.111 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:45.111 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:45.111 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:45.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:268: finishing write tid 1 to smithi10536496-29 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:268: finishing write tid 2 to smithi10536496-29 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:268: finishing write tid 3 to smithi10536496-29 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:268: finishing write tid 4 to smithi10536496-29 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:268: finishing write tid 5 to smithi10536496-29 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 27 (ObjNum 115 snap 22 seq_num 115) dirty exists 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:268: left oid 29 (ObjNum 115 snap 22 seq_num 115) 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:265: finishing copy_from racing read to smithi10536496-36 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:265: finishing copy_from to smithi10536496-36 2024-09-15T03:41:45.156 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 41 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 54 (ObjNum 85 snap 12 seq_num 85) dirty exists 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:264: done (4 left) 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:265: done (3 left) 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:266: done (2 left) 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:268: done (1 left) 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:271: done (0 left) 2024-09-15T03:41:45.157 INFO:tasks.rados.rados.0.smithi105.stdout:272: snap_create 2024-09-15T03:41:45.159 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:45.158+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:45.221 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:45.221 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:45.221 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:45.221 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:45.221 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:45.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:45.260 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:45.259+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:45.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-15T03:41:45.283+0000 7f8f507f1640 -1 mon.a@0(leader).osd e78 definitely_dead 0 2024-09-15T03:41:45.332 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:45.332 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.301574889000001, 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011155541, 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029084+0000", 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0603e-05 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029486+0000", 2024-09-15T03:41:45.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000402144 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029493+0000", 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.165e-06 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029683+0000", 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000190478 2024-09-15T03:41:45.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029706+0000", 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2849000000000001e-05 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040193+0000", 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010486822999999999 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040228+0000", 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4993000000000002e-05 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:41, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:45.338 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.300335943, 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098373709999999993, 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030312+0000", 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030333+0000", 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0635000000000001e-05 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030374+0000", 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030383+0000", 2024-09-15T03:41:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000004e-06 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030436+0000", 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2605000000000002e-05 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030447+0000", 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1576999999999999e-05 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040120+0000", 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096721959999999992 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040149+0000", 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9218e-05 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.299025248, 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013955543000000001, 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031623+0000", 2024-09-15T03:41:45.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031726+0000", 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103491 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031757+0000", 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0908999999999998e-05 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031770+0000", 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2565000000000001e-05 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031863+0000", 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2622999999999997e-05 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031928+0000", 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5984999999999999e-05 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045547+0000", 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013618655 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045578+0000", 2024-09-15T03:41:45.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0565999999999998e-05 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.298062076999997, 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013036834000000001, 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.349 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032586+0000", 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032605+0000", 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-05 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032648+0000", 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2763999999999999e-05 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032656+0000", 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032693+0000", 2024-09-15T03:41:45.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6701e-05 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032699+0000", 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6849999999999998e-06 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045592+0000", 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012893345 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045622+0000", 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0182e-05 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.295424633000003, 2024-09-15T03:41:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010405127, 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035224+0000", 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1800000000000005e-07 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035236+0000", 2024-09-15T03:41:45.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1952e-05 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035267+0000", 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035273+0000", 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035320+0000", 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6283999999999999e-05 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035326+0000", 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.105e-06 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045587+0000", 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010261562 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045628+0000", 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0469000000000002e-05 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=25, reqid=client.4562.0:101, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.246305728000003, 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011334317, 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:45.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084360+0000", 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7618999999999999e-05 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088093+0000", 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037333290000000001 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088098+0000", 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7010000000000004e-06 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088131+0000", 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2653000000000001e-05 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088136+0000", 2024-09-15T03:41:45.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4979999999999997e-06 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095632+0000", 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074962550000000003 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095676+0000", 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3516999999999997e-05 2024-09-15T03:41:45.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=27, reqid=client.4562.0:104, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.241811046999999, 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098078109999999996, 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088837+0000", 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088853+0000", 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5770999999999999e-05 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095603+0000", 2024-09-15T03:41:45.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067504870000000003 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095612+0000", 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4139999999999994e-06 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095655+0000", 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3417000000000001e-05 2024-09-15T03:41:45.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095665+0000", 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0006e-05 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098619+0000", 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029542119999999999 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098644+0000", 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4975999999999999e-05 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=32, to_read={3:edf86ef6:::smithi10536496-37:head=0,4096,0}, subchunks={3:edf86ef6:::smithi10536496-37:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.225207361999999, 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039281828999999997, 2024-09-15T03:41:45.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105441+0000", 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105447+0000", 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5010000000000001e-06 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141752+0000", 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036305044000000002 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141762+0000", 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3940000000000002e-06 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141935+0000", 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017359499999999999 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.144722+0000", 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00278671 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=34, reqid=client.4562.0:140, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.188134335000001, 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038895118999999999, 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142514+0000", 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.449e-05 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180934+0000", 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038406005999999999 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180939+0000", 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4270000000000001e-06 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180984+0000", 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5751000000000003e-05 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180995+0000", 2024-09-15T03:41:45.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0332e-05 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181383+0000", 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038812500000000002 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181408+0000", 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5349000000000001e-05 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=39, reqid=client.4562.0:163, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.145646523000003, 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099943480000000001, 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185002+0000", 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185014+0000", 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.205e-05 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194530+0000", 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095162920000000008 2024-09-15T03:41:45.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194535+0000", 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4090000000000002e-06 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194590+0000", 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4234000000000003e-05 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194595+0000", 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4369999999999998e-06 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194977+0000", 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038248600000000002 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194995+0000", 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799e-05 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=40, reqid=client.4562.0:176, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.136147043000001, 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229890967, 2024-09-15T03:41:45.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194501+0000", 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194514+0000", 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423889+0000", 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229374775 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423894+0000", 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0669999999999999e-06 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423938+0000", 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3967000000000001e-05 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423948+0000", 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2920000000000001e-06 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424370+0000", 2024-09-15T03:41:45.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042234599999999998 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424391+0000", 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1441999999999999e-05 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=74, reqid=client.4562.0:297, at_version=47'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.318365647, 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097576989999999999, 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.29e-07 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012308+0000", 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6588e-05 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012483+0000", 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017428699999999999 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012490+0000", 2024-09-15T03:41:45.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.013059+0000", 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056855499999999999 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.013070+0000", 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1286e-05 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.022002+0000", 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089322210000000006 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.022039+0000", 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7363999999999997e-05 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=75, reqid=client.4562.0:298, at_version=47'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.316385682, 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01362943, 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014263+0000", 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0100000000000001e-06 2024-09-15T03:41:45.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014284+0000", 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1742e-05 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014322+0000", 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7746000000000001e-05 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014330+0000", 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014378+0000", 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.799e-05 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014389+0000", 2024-09-15T03:41:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027856+0000", 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013467549000000001 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027891+0000", 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4925999999999997e-05 2024-09-15T03:41:45.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=76, reqid=client.4562.0:299, at_version=47'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.315311008999998, 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01257076, 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015337+0000", 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.65e-07 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015351+0000", 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4504e-05 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015447+0000", 2024-09-15T03:41:45.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5996999999999996e-05 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015454+0000", 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4250000000000003e-06 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016973+0000", 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001519248 2024-09-15T03:41:45.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016980+0000", 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4449999999999999e-06 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027896+0000", 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010915784 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027907+0000", 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0992000000000001e-05 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:399 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:45.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.279069545999999, 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014049582999999999, 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 399 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051580+0000", 2024-09-15T03:41:45.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-06 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051584+0000", 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3320000000000002e-06 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051594+0000", 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.027e-05 2024-09-15T03:41:45.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051600+0000", 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3500000000000002e-06 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:41:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5226000000000003e-05 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051663+0000", 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.25e-06 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058108+0000", 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064450269999999999 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058282+0000", 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017381800000000001 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059091+0000", 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080964600000000002 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062585+0000", 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003493875 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065538+0000", 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002952826 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065627+0000", 2024-09-15T03:41:45.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339000000000006e-05 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:400 3.15s0 3:ac8f3d24:::smithi10536496-17:head [rollback c] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.279033026, 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016394477000000001, 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 400 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051615+0000", 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051618+0000", 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.977e-06 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051623+0000", 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5010000000000004e-06 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051626+0000", 2024-09-15T03:41:45.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8009999999999999e-06 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051729+0000", 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010270800000000001 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051734+0000", 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059522+0000", 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007788059 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059592+0000", 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0457999999999995e-05 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060158+0000", 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056606199999999999 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062892+0000", 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002733769 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.067964+0000", 2024-09-15T03:41:45.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050720879999999998 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068009+0000", 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4685999999999999e-05 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:401 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:45.421 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.278999333000002, 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016504504, 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 401 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0100000000000001e-07 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051651+0000", 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:41:45.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051659+0000", 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051769+0000", 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011069 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051774+0000", 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6430000000000003e-06 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060449+0000", 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008674747 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060522+0000", 2024-09-15T03:41:45.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3393999999999996e-05 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060841+0000", 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031853799999999999 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062921+0000", 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002080765 2024-09-15T03:41:45.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068093+0000", 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051714500000000002 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068153+0000", 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9673000000000003e-05 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.278481771999999, 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37552997599999999, 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-07 2024-09-15T03:41:45.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.053528+0000", 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013622840000000001 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426011+0000", 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37248246099999999 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426019+0000", 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3089999999999996e-06 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426108+0000", 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9092999999999996e-05 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426118+0000", 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0407e-05 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427667+0000", 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015489220000000001 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427696+0000", 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8337999999999999e-05 2024-09-15T03:41:45.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:620 3.15s0 3:af19ecb4:::smithi10536496-23:28 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:43.014811+0000", 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.3158360849999999, 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010014596000000001, 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 620 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014811+0000", 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014811+0000", 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014812+0000", 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014817+0000", 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014822+0000", 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7370000000000003e-06 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014826+0000", 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6619999999999998e-06 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014982+0000", 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015647600000000001 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014989+0000", 2024-09-15T03:41:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2489999999999998e-06 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018063+0000", 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030738950000000001 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018125+0000", 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2119000000000006e-05 2024-09-15T03:41:45.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.024826+0000", 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067012599999999997 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:621 3.15s0 3:af19ecb4:::smithi10536496-23:28 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:45.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.3157917120000002, 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010658079000000001, 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 621 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0499999999999999e-07 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014859+0000", 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9440000000000001e-06 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014867+0000", 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.824e-06 2024-09-15T03:41:45.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014881+0000", 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4596000000000001e-05 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.015018+0000", 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013627299999999999 2024-09-15T03:41:45.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.015024+0000", 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7159999999999997e-06 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018326+0000", 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033014989999999998 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018382+0000", 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6098999999999999e-05 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.025514+0000", 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071317230000000004 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.446 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.447 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:45.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:45.448 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:45.447+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:45.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:45.549 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:45.548+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:45.552 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:45.552 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:45.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:45.580 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:45.581 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:45.602 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:45.602 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:45.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:45.623 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:45.643 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:45.644 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:45.650 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:45.649+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:45.662 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:45.663 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:45.663 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:45.663 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:45.663 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:45.663 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:45.663 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:45.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:45.682 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:45.682 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:45.703 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:45.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:45.723 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:45.724 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:45.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:45.743 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:45.751 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:45.751+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:45.763 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:45.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:45.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:45.784 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:45.804 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:45.804 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:45.816 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.965013360999997, 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:41:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:41:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:41:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.964985255999999, 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:41:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:41:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.670144149999999, 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:41:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:41:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:41:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:41:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=8, to_read={3:fff44eab:::smithi10536496-10:head=0,4096,0}, subchunks={3:fff44eab:::smithi10536496-10:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.780567806000001, 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046423195, 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033900+0000", 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033906+0000", 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080075+0000", 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046169038000000003 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080080+0000", 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2024-09-15T03:41:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080125+0000", 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5151000000000002e-05 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080322+0000", 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019686500000000001 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 35/32 ECSubWriteReply(tid=118, last_complete=35'2, committed=1, applied=1))", 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.727635524, 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021508349, 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086832+0000", 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086835+0000", 2024-09-15T03:41:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0479999999999999e-06 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108275+0000", 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021439848000000001 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108283+0000", 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8609999999999998e-06 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108320+0000", 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7330999999999999e-05 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108340+0000", 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9531999999999999e-05 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.706062158000002, 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041863826999999999, 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108419+0000", 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149106+0000", 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040686870999999999 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149110+0000", 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.335e-06 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149157+0000", 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7131e-05 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149164+0000", 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150245+0000", 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010804580000000001 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150269+0000", 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4128e-05 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:132 3.3s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.689252781, 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015465478, 2024-09-15T03:41:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 132 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125215+0000", 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125216+0000", 2024-09-15T03:41:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7349999999999999e-06 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125223+0000", 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125227+0000", 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8840000000000004e-06 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129903+0000", 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046767349999999996 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129995+0000", 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1706000000000006e-05 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133187+0000", 2024-09-15T03:41:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031916510000000002 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133234+0000", 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7862999999999999e-05 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140680+0000", 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074451860000000003 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=25, to_read={3:fc38c57e:::smithi10536496-49:head=0,4096,0}, subchunks={3:fc38c57e:::smithi10536496-49:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.658012018999997, 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038157982, 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156460+0000", 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194316+0000", 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037855609999999998 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194320+0000", 2024-09-15T03:41:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999998e-06 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194361+0000", 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0708999999999999e-05 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194613+0000", 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025227700000000002 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=31, to_read={3:f17f8228:::smithi10536496-40:head=0,167936,0}, subchunks={3:f17f8228:::smithi10536496-40:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.618563829000003, 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066290004, 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195904+0000", 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195911+0000", 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3030000000000002e-06 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262021+0000", 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066109629000000003 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262025+0000", 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5909999999999998e-06 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262085+0000", 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0183999999999998e-05 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262193+0000", 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107648 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:277 3.13s0 3:ca2bd2fa:::smithi10536496-45:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.784912790999996, 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0157117019999999, 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 277 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029556+0000", 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.542e-06 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029563+0000", 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2989999999999999e-06 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029577+0000", 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029601+0000", 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3200000000000001e-05 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029661+0000", 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0633e-05 2024-09-15T03:41:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029665+0000", 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5990000000000002e-06 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037509+0000", 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0078444609999999 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037649+0000", 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139476 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.045266+0000", 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076171729999999997 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.5s3 47/44 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=47'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-15T03:41:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.811291149999999, 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013250879, 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4600000000000001e-07 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003206+0000", 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9589000000000001e-05 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004002+0000", 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079689099999999998 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004013+0000", 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004302+0000", 2024-09-15T03:41:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000288554 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004314+0000", 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2277e-05 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016390+0000", 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012076 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016427+0000", 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6909000000000002e-05 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:305 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.809974916000002, 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0410521429999999, 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 305 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004494+0000", 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1660000000000001e-06 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004502+0000", 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1540000000000002e-06 2024-09-15T03:41:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004520+0000", 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7742000000000001e-05 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004530+0000", 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004681+0000", 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015092999999999999 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004692+0000", 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1232e-05 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009082+0000", 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043904649999999996 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009087+0000", 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7339999999999999e-06 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.033828+0000", 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0247413160000001 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.034013+0000", 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018443999999999999 2024-09-15T03:41:45.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045544+0000", 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011531351 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:306 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.809878736999998, 2024-09-15T03:41:45.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0413215819999999, 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 306 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004590+0000", 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3260000000000001e-06 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004596+0000", 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010000000000002e-06 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004613+0000", 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6892999999999999e-05 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004623+0000", 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268000000000001e-05 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009818+0000", 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051952880000000002 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009821+0000", 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0189999999999998e-06 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035613+0000", 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.025791608 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035700+0000", 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7298000000000003e-05 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045910+0000", 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010209480999999999 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:307 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.809834822999999, 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.041622225, 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 307 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004633+0000", 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0200000000000001e-07 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004639+0000", 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5599999999999999e-06 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004649+0000", 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004656+0000", 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8870000000000003e-06 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051891749999999999 2024-09-15T03:41:45.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035995+0000", 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026149322 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.036088+0000", 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3162e-05 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.046254+0000", 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010166405999999999 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:335 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:45.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.693666785000001, 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077783689000000003, 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 335 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120802+0000", 2024-09-15T03:41:45.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120808+0000", 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2999999999999998e-06 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120821+0000", 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120826+0000", 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7879999999999997e-06 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120884+0000", 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8034e-05 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.121005+0000", 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120637 2024-09-15T03:41:45.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.130702+0000", 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096976079999999999 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.132697+0000", 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019945800000000001 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198536+0000", 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065839133999999994 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198584+0000", 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7796000000000001e-05 2024-09-15T03:41:45.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:369 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~589824] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.789188407000001, 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022704654000000001, 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 369 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025281+0000", 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4159999999999998e-06 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028105+0000", 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028242380000000002 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028126+0000", 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0939999999999999e-05 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028134+0000", 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7649999999999993e-06 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032973+0000", 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048395670000000003 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.033153+0000", 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017928700000000001 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034716+0000", 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001562873 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047854+0000", 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013138212999999999 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047923+0000", 2024-09-15T03:41:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9306000000000004e-05 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047983+0000", 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0049000000000001e-05 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:370 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~573440] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:41:45.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.781503108999999, 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018633602999999999, 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 370 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:41:45.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032966+0000", 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034471+0000", 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015050790000000001 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034522+0000", 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0529999999999999e-05 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034528+0000", 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1750000000000002e-06 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041526+0000", 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006997886 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041652+0000", 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126415 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.042320+0000", 2024-09-15T03:41:45.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066831300000000002 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051454+0000", 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091331659999999999 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051523+0000", 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9029999999999995e-05 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051597+0000", 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4819000000000003e-05 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:371 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~196608] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.779928967, 2024-09-15T03:41:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827797000000001, 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 371 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034539+0000", 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.116e-06 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034715+0000", 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017624599999999999 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034727+0000", 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1912999999999999e-05 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034736+0000", 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:41:45.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044278+0000", 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095422289999999993 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044402+0000", 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124227 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.045119+0000", 2024-09-15T03:41:45.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071647099999999999 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054221+0000", 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091019350000000002 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054291+0000", 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0148999999999995e-05 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054366+0000", 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4883999999999997e-05 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:372 3.18s0 3:1c642a04:::smithi10536496-46:head [setxattr _header (56) in=7b] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.779722453000002, 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020478778, 2024-09-15T03:41:45.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 372 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4199999999999995e-07 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034749+0000", 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034759+0000", 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4409999999999999e-06 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034769+0000", 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0064999999999999e-05 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047175+0000", 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012405657 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047305+0000", 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013068800000000001 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047652+0000", 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034670499999999999 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054413+0000", 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067614540000000001 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055174+0000", 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076077999999999996 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055223+0000", 2024-09-15T03:41:45.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9076999999999998e-05 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:373 3.18s0 3:1c642a04:::smithi10536496-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.779692525000002, 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021752218, 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:45.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 373 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:41:45.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034775+0000", 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034778+0000", 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9249999999999999e-06 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034785+0000", 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5669999999999997e-06 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034790+0000", 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.51e-06 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048073+0000", 2024-09-15T03:41:45.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013283497 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048128+0000", 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4455e-05 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055532+0000", 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074043889999999999 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055612+0000", 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9971999999999999e-05 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.056527+0000", 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091438399999999997 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:45.927 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:45.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:45.929 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:45.928+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:45.929 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:45.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:45.842+0000 7f57037fe640 0 -- 172.21.15.5:0/2199484010 >> [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] conn(0x7f5704188f90 msgr2=0x7f570418b370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:45.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:46.029 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:46.028+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:46.038 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.038 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:46.118 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.4. 2024-09-15T03:41:46.119 INFO:tasks.rados.rados.0.smithi105.stdout:272: done (0 left) 2024-09-15T03:41:46.119 INFO:tasks.rados.rados.0.smithi105.stdout:273: read oid 38 snap -1 2024-09-15T03:41:46.119 INFO:tasks.rados.rados.0.smithi105.stdout:273: expect deleted 2024-09-15T03:41:46.119 INFO:tasks.rados.rados.0.smithi105.stdout:274: snap_remove snap 21 2024-09-15T03:41:46.130 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:46.129+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:46.150 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.150 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.159 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:46.231 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:46.230+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:46.258 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.258 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.267 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:46.332 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:46.331+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:46.364 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.365 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:46.433 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:46.431+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:46.511 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.511 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:46.533 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:46.532+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:46.616 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.616 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:46.634 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:46.634+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:46.722 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.723 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:46.735 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:46.734+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:46.763 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:46.785 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:46.785 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:46.808 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:46.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:46.829 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.829 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:46.862 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:46.860+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:46.935 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:46.935 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:46.937 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:46.937+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:46.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:47.039 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:47.038+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:47.041 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.041 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:41:47.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:47.125 INFO:tasks.rados.rados.0.smithi105.stdout:274: done (1 left) 2024-09-15T03:41:47.125 INFO:tasks.rados.rados.0.smithi105.stdout:275: copy_from oid 42 from oid 35 current snap is 23 2024-09-15T03:41:47.125 INFO:tasks.rados.rados.0.smithi105.stdout:276: read oid 36 snap 18 2024-09-15T03:41:47.126 INFO:tasks.rados.rados.0.smithi105.stdout:276: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:47.126 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 37 current snap is 23 2024-09-15T03:41:47.126 INFO:tasks.rados.rados.0.smithi105.stdout:277: seq_num 116 ranges {1277952=786432,2064384=458752,2523136=442368,2965504=262144} 2024-09-15T03:41:47.138 INFO:tasks.rados.rados.0.smithi105.stdout:277: writing smithi10536496-37 from 1277952 to 2064384 tid 1 2024-09-15T03:41:47.141 INFO:tasks.rados.rados.0.smithi105.stdout:277: writing smithi10536496-37 from 2064384 to 2523136 tid 2 2024-09-15T03:41:47.141 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:47.139+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:47.143 INFO:tasks.rados.rados.0.smithi105.stdout:277: writing smithi10536496-37 from 2523136 to 2965504 tid 3 2024-09-15T03:41:47.145 INFO:tasks.rados.rados.0.smithi105.stdout:277: writing smithi10536496-37 from 2965504 to 3227648 tid 4 2024-09-15T03:41:47.145 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 10 current snap is 23 2024-09-15T03:41:47.145 INFO:tasks.rados.rados.0.smithi105.stdout:278: seq_num 117 ranges {1425408=425984,1851392=606208,2457600=540672,2998272=409600} 2024-09-15T03:41:47.156 INFO:tasks.rados.rados.0.smithi105.stdout:278: writing smithi10536496-10 from 1425408 to 1851392 tid 1 2024-09-15T03:41:47.160 INFO:tasks.rados.rados.0.smithi105.stdout:278: writing smithi10536496-10 from 1851392 to 2457600 tid 2 2024-09-15T03:41:47.163 INFO:tasks.rados.rados.0.smithi105.stdout:278: writing smithi10536496-10 from 2457600 to 2998272 tid 3 2024-09-15T03:41:47.167 INFO:tasks.rados.rados.0.smithi105.stdout:278: writing smithi10536496-10 from 2998272 to 3407872 tid 4 2024-09-15T03:41:47.167 INFO:tasks.rados.rados.0.smithi105.stdout:279: read oid 16 snap -1 2024-09-15T03:41:47.167 INFO:tasks.rados.rados.0.smithi105.stdout:279: expect deleted 2024-09-15T03:41:47.167 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 11 current snap is 23 2024-09-15T03:41:47.167 INFO:tasks.rados.rados.0.smithi105.stdout:280: seq_num 118 ranges {1769472=442368,2211840=753664,2965504=655360,3620864=147456} 2024-09-15T03:41:47.179 INFO:tasks.rados.rados.0.smithi105.stdout:280: writing smithi10536496-11 from 1769472 to 2211840 tid 1 2024-09-15T03:41:47.183 INFO:tasks.rados.rados.0.smithi105.stdout:280: writing smithi10536496-11 from 2211840 to 2965504 tid 2 2024-09-15T03:41:47.187 INFO:tasks.rados.rados.0.smithi105.stdout:280: writing smithi10536496-11 from 2965504 to 3620864 tid 3 2024-09-15T03:41:47.188 INFO:tasks.rados.rados.0.smithi105.stdout:280: writing smithi10536496-11 from 3620864 to 3768320 tid 4 2024-09-15T03:41:47.188 INFO:tasks.rados.rados.0.smithi105.stdout:281: read oid 28 snap -1 2024-09-15T03:41:47.188 INFO:tasks.rados.rados.0.smithi105.stdout:281: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:47.188 INFO:tasks.rados.rados.0.smithi105.stdout:282: read oid 21 snap -1 2024-09-15T03:41:47.189 INFO:tasks.rados.rados.0.smithi105.stdout:282: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T03:41:47.189 INFO:tasks.rados.rados.0.smithi105.stdout:283: snap_remove snap 22 2024-09-15T03:41:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:47.241 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:47.240+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:47.307 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.308 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.316 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:47.342 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:47.341+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:47.417 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.417 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.426 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:47.442 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:47.441+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:47.522 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.523 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.532 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:47.575 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:47.574+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:47.629 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.629 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.639 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:47.644 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:47.643+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:47.745 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:47.744+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:47.777 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.777 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.785 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:47.846 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:47.845+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:47.882 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.882 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.890 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:47.946 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:47.946+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:47.987 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:47.987 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:47.995 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:48.047 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:48.046+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:48.093 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.093 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.101 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:48.130 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:41:48.140 INFO:tasks.rados.rados.0.smithi105.stdout:283: done (9 left) 2024-09-15T03:41:48.140 INFO:tasks.rados.rados.0.smithi105.stdout:284: rollback oid 1 current snap is 23 2024-09-15T03:41:48.140 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 1 to 18 2024-09-15T03:41:48.140 INFO:tasks.rados.rados.0.smithi105.stdout:276: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:48.186 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:48.185+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:48.249 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.249 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.250 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:48.248+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:48.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:48.276 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing write tid 1 to smithi10536496-37 2024-09-15T03:41:48.276 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing write tid 2 to smithi10536496-37 2024-09-15T03:41:48.276 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing write tid 3 to smithi10536496-37 2024-09-15T03:41:48.276 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing write tid 4 to smithi10536496-37 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing write tid 5 to smithi10536496-37 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing write tid 6 to smithi10536496-37 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 46 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:277: left oid 37 (ObjNum 116 snap 23 seq_num 116) 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing write tid 1 to smithi10536496-10 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing write tid 2 to smithi10536496-10 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing write tid 3 to smithi10536496-10 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:276: done (9 left) 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:277: done (8 left) 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:285: read oid 8 snap -1 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:285: expect (ObjNum 85 snap 12 seq_num 85) 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:286: rollback oid 14 current snap is 23 2024-09-15T03:41:48.277 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 14 to 18 2024-09-15T03:41:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing write tid 4 to smithi10536496-10 2024-09-15T03:41:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing write tid 5 to smithi10536496-10 2024-09-15T03:41:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing write tid 6 to smithi10536496-10 2024-09-15T03:41:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 58 (ObjNum 117 snap 23 seq_num 117) dirty exists 2024-09-15T03:41:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:278: left oid 10 (ObjNum 117 snap 23 seq_num 117) 2024-09-15T03:41:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:282: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-15T03:41:48.287 INFO:tasks.rados.rados.0.smithi105.stdout:281: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:48.299 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing write tid 1 to smithi10536496-11 2024-09-15T03:41:48.299 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing write tid 2 to smithi10536496-11 2024-09-15T03:41:48.299 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing write tid 3 to smithi10536496-11 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:278: done (9 left) 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:279: done (8 left) 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:281: done (7 left) 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:282: done (6 left) 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:287: setattr oid 34 current snap is 23 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing write tid 4 to smithi10536496-11 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing write tid 5 to smithi10536496-11 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing write tid 6 to smithi10536496-11 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 67 (ObjNum 118 snap 23 seq_num 118) dirty exists 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:280: left oid 11 (ObjNum 118 snap 23 seq_num 118) 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:284: finishing rollback tid 0 to smithi10536496-1 2024-09-15T03:41:48.300 INFO:tasks.rados.rados.0.smithi105.stdout:284: finishing rollback tid 1 to smithi10536496-1 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:284: finishing rollback tid 2 to smithi10536496-1 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 54 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:275: finishing copy_from racing read to smithi10536496-42 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:275: finishing copy_from to smithi10536496-42 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:286: finishing rollback tid 0 to smithi10536496-14 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:286: finishing rollback tid 1 to smithi10536496-14 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:286: finishing rollback tid 2 to smithi10536496-14 2024-09-15T03:41:48.301 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 52 (ObjNum 77 snap 8 seq_num 77) dirty exists 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:273: done (6 left) 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:275: done (5 left) 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:280: done (4 left) 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:284: done (3 left) 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:286: done (2 left) 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:288: delete oid 38 current snap is 23 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:288: done (2 left) 2024-09-15T03:41:48.302 INFO:tasks.rados.rados.0.smithi105.stdout:289: delete oid 49 current snap is 23 2024-09-15T03:41:48.304 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 36 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-15T03:41:48.306 INFO:tasks.rados.rados.0.smithi105.stdout:285: expect (ObjNum 85 snap 12 seq_num 85) 2024-09-15T03:41:48.351 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:48.350+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:48.375 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.375 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.384 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:48.452 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:48.451+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:48.522 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.522 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.530 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:41:48.574 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:48.573+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:48.627 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.628 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.636 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:41:48.653 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:48.652+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:48.754 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:48.753+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:48.772 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.772 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:48.789 INFO:tasks.rados.rados.0.smithi105.stdout:285: done (2 left) 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:287: done (1 left) 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:289: done (0 left) 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:290: read oid 36 snap 18 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:290: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:291: read oid 27 snap -1 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:291: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:292: copy_from oid 17 from oid 2 current snap is 23 2024-09-15T03:41:48.790 INFO:tasks.rados.rados.0.smithi105.stdout:293: snap_create 2024-09-15T03:41:48.794 INFO:tasks.rados.rados.0.smithi105.stdout:291: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-15T03:41:48.807 INFO:tasks.rados.rados.0.smithi105.stdout:290: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:48.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:48.856 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:48.854+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:48.898 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:48.898 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:48.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:48.926 INFO:tasks.rados.rados.0.smithi105.stdout:292: finishing copy_from racing read to smithi10536496-17 2024-09-15T03:41:48.926 INFO:tasks.rados.rados.0.smithi105.stdout:292: finishing copy_from to smithi10536496-17 2024-09-15T03:41:48.926 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 39 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-15T03:41:48.956 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:48.956+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:48.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:48.984 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:48.984 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:49.005 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:49.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:49.014 INFO:tasks.rados.rados.0.smithi105.stdout:290: done (3 left) 2024-09-15T03:41:49.014 INFO:tasks.rados.rados.0.smithi105.stdout:291: done (2 left) 2024-09-15T03:41:49.014 INFO:tasks.rados.rados.0.smithi105.stdout:292: done (1 left) 2024-09-15T03:41:49.014 INFO:tasks.rados.rados.0.smithi105.stdout:293: done (0 left) 2024-09-15T03:41:49.014 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 22 current snap is 24 2024-09-15T03:41:49.014 INFO:tasks.rados.rados.0.smithi105.stdout:294: seq_num 121 ranges {0=491520,491520=802816,1294336=720896,2015232=32768} 2024-09-15T03:41:49.016 INFO:tasks.rados.rados.0.smithi105.stdout:294: writing smithi10536496-22 from 0 to 491520 tid 1 2024-09-15T03:41:49.020 INFO:tasks.rados.rados.0.smithi105.stdout:294: writing smithi10536496-22 from 491520 to 1294336 tid 2 2024-09-15T03:41:49.022 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:49.022 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:49.022 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:49.024 INFO:tasks.rados.rados.0.smithi105.stdout:294: writing smithi10536496-22 from 1294336 to 2015232 tid 3 2024-09-15T03:41:49.025 INFO:tasks.rados.rados.0.smithi105.stdout:294: writing smithi10536496-22 from 2015232 to 2048000 tid 4 2024-09-15T03:41:49.025 INFO:tasks.rados.rados.0.smithi105.stdout:295: copy_from oid 25 from oid 31 current snap is 24 2024-09-15T03:41:49.025 INFO:tasks.rados.rados.0.smithi105.stdout:296: rollback oid 37 current snap is 24 2024-09-15T03:41:49.025 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 37 to 23 2024-09-15T03:41:49.025 INFO:tasks.rados.rados.0.smithi105.stdout:297: copy_from oid 39 from oid 42 current snap is 24 2024-09-15T03:41:49.025 INFO:tasks.rados.rados.0.smithi105.stdout:298: snap_create 2024-09-15T03:41:49.031 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:49.031 INFO:tasks.rados.rados.0.smithi105.stdout:296: finishing rollback tid 0 to smithi10536496-37 2024-09-15T03:41:49.032 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:49.032 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:49.033 INFO:tasks.rados.rados.0.smithi105.stdout:296: finishing rollback tid 1 to smithi10536496-37 2024-09-15T03:41:49.033 INFO:tasks.rados.rados.0.smithi105.stdout:296: finishing rollback tid 2 to smithi10536496-37 2024-09-15T03:41:49.034 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 56 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-15T03:41:49.051 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:49.052 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:49.057 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:49.056+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:49.072 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:49.135 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:49.135 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:49.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:7 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.116420269999999, 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01304598, 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.136 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 7 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017045+0000", 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5040000000000001e-06 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017174+0000", 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012955 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017203+0000", 2024-09-15T03:41:49.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8303e-05 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017222+0000", 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9680000000000001e-05 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021260800000000001 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017608+0000", 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017282699999999999 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018409+0000", 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080141399999999997 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020887+0000", 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002477244 2024-09-15T03:41:49.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030008+0000", 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0091209519999999999 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030088+0000", 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0897999999999997e-05 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.141 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:8 3.6s0 3:62396a93:::smithi10536496-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.116206617000003, 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013086021999999999, 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 8 2024-09-15T03:41:49.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017258+0000", 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6300000000000001e-06 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017267+0000", 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4830000000000005e-06 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017303+0000", 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5326e-05 2024-09-15T03:41:49.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017316+0000", 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3073e-05 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019094+0000", 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001778352 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0492e-05 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019834+0000", 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064979700000000003 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021641+0000", 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018066779999999999 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030253+0000", 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086118240000000006 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030342+0000", 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9367e-05 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:9 3.6s0 3:62396a93:::smithi10536496-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.116072985999999, 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015701551000000001, 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 9 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017391+0000", 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.23e-07 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017394+0000", 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5999999999999998e-06 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017406+0000", 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1980999999999999e-05 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8459e-05 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020095+0000", 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026602399999999999 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020181+0000", 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6468e-05 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030742+0000", 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010561054 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015357899999999999 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.033091+0000", 2024-09-15T03:41:49.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021953469999999998 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.111931560000002, 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050576110000000001, 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021532+0000", 2024-09-15T03:41:49.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021586+0000", 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4029000000000001e-05 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023870+0000", 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022843490000000002 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023879+0000", 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7509999999999997e-06 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024332+0000", 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00045252999999999998 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024353+0000", 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1382999999999998e-05 2024-09-15T03:41:49.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026560+0000", 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002206679 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026589+0000", 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9149000000000002e-05 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:79 3.6s0 3:60903cd9:::smithi10536496-27:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.074283803999997, 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056797210000000004, 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 79 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.158 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059181+0000", 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.905e-06 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059725+0000", 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000544675 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059743+0000", 2024-09-15T03:41:49.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059749+0000", 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8680000000000001e-06 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059823+0000", 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3856999999999994e-05 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059937+0000", 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113934 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.060672+0000", 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00073485 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062007+0000", 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001335433 2024-09-15T03:41:49.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064787+0000", 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027791140000000001 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064859+0000", 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2015999999999996e-05 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:80 3.6s0 3:60903cd9:::smithi10536496-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.073681489000002, 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052634420000000001, 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 80 2024-09-15T03:41:49.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059782+0000", 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059787+0000", 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9010000000000002e-06 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059797+0000", 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:41:49.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059802+0000", 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7310000000000001e-06 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061262+0000", 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014593609999999999 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061347+0000", 2024-09-15T03:41:49.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5358999999999994e-05 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061756+0000", 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00040932300000000001 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062312+0000", 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055575999999999998 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064973+0000", 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026609950000000002 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065045+0000", 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1703000000000001e-05 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:81 3.6s0 3:60903cd9:::smithi10536496-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.073637224999999, 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055321019999999997, 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.170 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 81 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:49.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059826+0000", 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4899999999999995e-07 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059830+0000", 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5769999999999998e-06 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059837+0000", 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4490000000000002e-06 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059841+0000", 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.765e-06 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062037+0000", 2024-09-15T03:41:49.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002196506 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062113+0000", 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5740000000000003e-05 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065347+0000", 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003233648 2024-09-15T03:41:49.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065358+0000", 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0868e-05 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=60, reqid=client.4562.0:281, at_version=45'25, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:49.175 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 37.111698564999998, 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068718160000000002, 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4499999999999999e-07 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021848+0000", 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4214000000000004e-05 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024178+0000", 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023298640000000001 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024186+0000", 2024-09-15T03:41:49.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3379999999999999e-06 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024242+0000", 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5829999999999999e-05 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024252+0000", 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028612+0000", 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043595680000000003 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028636+0000", 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3896999999999998e-05 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.180 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 37.108218487000002, 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087088040000000005, 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025245+0000", 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0600000000000001e-07 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025397+0000", 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015260800000000001 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026753+0000", 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013562590000000001 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026763+0000", 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7340000000000003e-06 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026806+0000", 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3254000000000003e-05 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026816+0000", 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0115000000000001e-05 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033305+0000", 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00648871 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033953+0000", 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064781799999999998 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:49.185 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 37.105740992000001, 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006712165, 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027799+0000", 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6990999999999995e-05 2024-09-15T03:41:49.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028647+0000", 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00084766699999999997 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028655+0000", 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6769999999999995e-06 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028695+0000", 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0281000000000003e-05 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028705+0000", 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7140000000000005e-06 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034407+0000", 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.005702141 2024-09-15T03:41:49.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034434+0000", 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7005e-05 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:49.190 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 37.105660090000001, 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066515380000000002, 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:49.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027818+0000", 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4961e-05 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029147+0000", 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001328842 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029156+0000", 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8710000000000003e-06 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029196+0000", 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9928999999999997e-05 2024-09-15T03:41:49.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029204+0000", 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5809999999999997e-06 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034438+0000", 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052331399999999998 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034454+0000", 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6558e-05 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(osd.14.22:2 3.6s0 3:62396a93:::smithi10536496-3:head [assert-version v8,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:49.195 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 37.089957206000001, 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010013731999999999, 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043507+0000", 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9290000000000001e-06 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043518+0000", 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0934000000000001e-05 2024-09-15T03:41:49.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043540+0000", 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1418000000000001e-05 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043548+0000", 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.602e-06 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043611+0000", 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3034000000000007e-05 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043784+0000", 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017247700000000001 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.053519+0000", 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097353379999999996 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.199 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:549 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~671744] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.121786+0000", 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.011676968, 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097161899999999995, 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.200 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 549 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.121786+0000", 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.121786+0000", 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.121788+0000", 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.272e-06 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122452+0000", 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000664056 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122475+0000", 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3404000000000002e-05 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122483+0000", 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2710000000000003e-06 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122539+0000", 2024-09-15T03:41:49.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6713999999999999e-05 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122615+0000", 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5593999999999995e-05 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.123438+0000", 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082271199999999997 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.124658+0000", 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001219873 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131436+0000", 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067785520000000002 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131502+0000", 2024-09-15T03:41:49.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5741999999999995e-05 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:550 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~753664] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.125979+0000", 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.007483492, 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011372897, 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.206 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 550 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.125979+0000", 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.125979+0000", 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.125981+0000", 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.674e-06 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126650+0000", 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00066883899999999996 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126664+0000", 2024-09-15T03:41:49.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4481000000000001e-05 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126671+0000", 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4359999999999999e-06 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126708+0000", 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7712000000000001e-05 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126782+0000", 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4101000000000001e-05 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.127462+0000", 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067922600000000001 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128515+0000", 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001053567 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137298+0000", 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087826050000000006 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137352+0000", 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4255999999999997e-05 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:551 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~409600] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128415+0000", 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.005047425000001, 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090658130000000007, 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 551 2024-09-15T03:41:49.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128415+0000", 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128415+0000", 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128417+0000", 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2950000000000001e-06 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128715+0000", 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00029882200000000002 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128731+0000", 2024-09-15T03:41:49.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5769000000000001e-05 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128737+0000", 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3289999999999999e-06 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128785+0000", 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7641e-05 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128908+0000", 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012250499999999999 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.129698+0000", 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00079081699999999997 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131104+0000", 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001405534 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137443+0000", 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063393989999999999 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137481+0000", 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7701999999999999e-05 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:552 3.6s0 3:62396a93:::smithi10536496-3:head [setxattr _header (56) in=7b] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128748+0000", 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.004714528999999, 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56563410999999997, 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 552 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128748+0000", 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128748+0000", 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128749+0000", 2024-09-15T03:41:49.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0800000000000004e-07 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128753+0000", 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2849999999999996e-06 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128759+0000", 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6670000000000001e-06 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128762+0000", 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7989999999999998e-06 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.130334+0000", 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015723379999999999 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.130431+0000", 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6822000000000002e-05 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.130812+0000", 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038130299999999999 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131518+0000", 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00070572400000000002 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694297+0000", 2024-09-15T03:41:49.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.562779221 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694382+0000", 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5242999999999997e-05 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:553 3.6s0 3:62396a93:::smithi10536496-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:49.223 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.004697819, 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.566828946, 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 553 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128767+0000", 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.511e-06 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128770+0000", 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5700000000000001e-06 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128772+0000", 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.181e-06 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131204+0000", 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024313859999999998 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131295+0000", 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0939000000000005e-05 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694733+0000", 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56343809300000003 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694791+0000", 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8378999999999998e-05 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.695594+0000", 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080257799999999997 2024-09-15T03:41:49.228 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 71/44 ECSubWriteReply(tid=29, last_complete=71'16, committed=1, applied=1))", 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 10.999439162, 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56068105000000001, 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:49.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.13e-07 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134026+0000", 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3810000000000002e-06 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694158+0000", 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56013147900000004 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694162+0000", 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694237+0000", 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4108000000000003e-05 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694705+0000", 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00046806300000000001 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.13s4 78/72 ECSubWrite(tid=147, reqid=client.4562.0:642, at_version=78'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.1021014689999999, 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049401810000000001, 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:49.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.55e-07 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031444+0000", 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2619000000000003e-05 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031592+0000", 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014789800000000001 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031601+0000", 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031667+0000", 2024-09-15T03:41:49.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5809000000000006e-05 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031677+0000", 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9690000000000006e-06 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036275+0000", 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045977190000000001 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036301+0000", 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6818e-05 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 78/32 ECSubWrite(tid=149, reqid=osd.11.0:149, at_version=78'44, trim_to=0'0, roll_forward_to=77'40))", 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.101989508, 2024-09-15T03:41:49.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055448110000000002, 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031492+0000", 2024-09-15T03:41:49.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8856e-05 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032145+0000", 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00065225899999999998 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032159+0000", 2024-09-15T03:41:49.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4798e-05 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.033418+0000", 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00125884 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.033428+0000", 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9750000000000002e-06 2024-09-15T03:41:49.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036990+0000", 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003561397 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.037018+0000", 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8393000000000001e-05 2024-09-15T03:41:49.243 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.244 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.244 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:49.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:49.246 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:49.244+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:49.293 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:49.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:49.313 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:49.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:49.347 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:49.348 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:49.351 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:49.349+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:49.382 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:49.383 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:49.410 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:49.417 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:49.417 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:49.417 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:49.417 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:49.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:49.430 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:49.430 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:49.449 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:49.450 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:49.451 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:49.450+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:49.470 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:49.470 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:49.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:49.469+0000 7f08fffff640 0 -- 172.21.15.5:0/3716895112 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f09001930a0 msgr2=0x7f0900195480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:49.533 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:49.533 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:49.534 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:49.534 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:49.534 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:49.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:49.548 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:49.547+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:49.648 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:49.648+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:49.686 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:101 3.bs0 3:db73af7f:::smithi10536496-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.602188038000001, 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060658139999999999, 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 101 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082525+0000", 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082528+0000", 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0989999999999999e-06 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082540+0000", 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2673999999999999e-05 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082544+0000", 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9559999999999999e-06 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083692+0000", 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011471949999999999 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083836+0000", 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014488899999999999 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084281+0000", 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044464500000000002 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085132+0000", 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00085125900000000004 2024-09-15T03:41:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143106+0000", 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057973867999999998 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143182+0000", 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5862e-05 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:102 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.602146427999998, 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100342765, 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 102 2024-09-15T03:41:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082569+0000", 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2279999999999998e-06 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082577+0000", 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9319999999999996e-06 2024-09-15T03:41:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082584+0000", 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8009999999999998e-06 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084479+0000", 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018943709999999999 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084576+0000", 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7615000000000006e-05 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143227+0000", 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058651048999999997 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143302+0000", 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4547000000000003e-05 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182908+0000", 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039606548999999998 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=25, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.599054254000002, 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057539988, 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.98e-07 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085660+0000", 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.902e-06 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142986+0000", 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057326329000000002 2024-09-15T03:41:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142992+0000", 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143038+0000", 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6576999999999998e-05 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143198+0000", 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015933200000000001 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:111 3.7s0 3:edf86ef6:::smithi10536496-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.590330979999997, 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050567135999999999, 2024-09-15T03:41:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 111 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094382+0000", 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094384+0000", 2024-09-15T03:41:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7489999999999999e-06 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094393+0000", 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2500000000000006e-06 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094397+0000", 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8040000000000001e-06 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002394565 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096872+0000", 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0492000000000001e-05 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105071+0000", 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081983339999999998 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105164+0000", 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3202999999999995e-05 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.144948+0000", 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039784183000000001 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:141 3.bs0 3:dc608269:::smithi10536496-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.544707948000003, 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055455817999999997, 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 141 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6600000000000003e-07 2024-09-15T03:41:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140006+0000", 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5409999999999999e-06 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140011+0000", 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.843e-06 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140013+0000", 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.621e-06 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002515181 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142605+0000", 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6328999999999994e-05 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182018+0000", 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039413493000000001 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182116+0000", 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8159000000000007e-05 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.195460+0000", 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013343384999999999 2024-09-15T03:41:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:273 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~802816] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 38.657793904000002, 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0316972600000001, 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026920+0000", 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5629999999999999e-06 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027741+0000", 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082160000000000002 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027764+0000", 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2495000000000001e-05 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027772+0000", 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027837+0000", 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5062999999999997e-05 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027841+0000", 2024-09-15T03:41:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043211+0000", 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.015370181 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043389+0000", 2024-09-15T03:41:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017807500000000001 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044733+0000", 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013438510000000001 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048847+0000", 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041134250000000004 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058556+0000", 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097088929999999997 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058615+0000", 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9623999999999997e-05 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:274 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~376832] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 38.655313829000001, 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.029738034, 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 274 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029399+0000", 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6099999999999999e-07 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029738+0000", 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00033941400000000001 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029756+0000", 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524e-05 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029763+0000", 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9609999999999999e-06 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029831+0000", 2024-09-15T03:41:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8169000000000004e-05 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029833+0000", 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045818+0000", 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0159847550000001 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045920+0000", 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102085 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.046687+0000", 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00076710099999999996 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049023+0000", 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023356919999999999 2024-09-15T03:41:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059069+0000", 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010046019999999999 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059136+0000", 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7156999999999994e-05 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:275 3.17s0 3:ec08d906:::smithi10536496-14:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 38.654921944999998, 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0294973439999999, 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 275 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029791+0000", 2024-09-15T03:41:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029807+0000", 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6107e-05 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029828+0000", 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1475999999999999e-05 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029838+0000", 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9879999999999999e-06 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029885+0000", 2024-09-15T03:41:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6887e-05 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029887+0000", 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5030000000000001e-06 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047307+0000", 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0174202429999999 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047434+0000", 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012744699999999999 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048033+0000", 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00059865199999999997 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049643+0000", 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016096159999999999 2024-09-15T03:41:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059241+0000", 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095980579999999996 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059287+0000", 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6726e-05 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:276 3.17s0 3:ec08d906:::smithi10536496-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 38.654855384000001, 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0302367509999999, 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 276 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029862+0000", 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4259999999999996e-06 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029875+0000", 2024-09-15T03:41:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3132999999999999e-05 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029880+0000", 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029913+0000", 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3079e-05 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029914+0000", 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.164e-06 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048304+0000", 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.018390516 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048405+0000", 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010111300000000001 2024-09-15T03:41:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059337+0000", 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010931523 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059408+0000", 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0962000000000003e-05 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.060093+0000", 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068532499999999997 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.684033462000002, 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.030189832, 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000680+0000", 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000683+0000", 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6610000000000002e-06 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000708+0000", 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5165999999999999e-05 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000714+0000", 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6250000000000004e-06 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005624+0000", 2024-09-15T03:41:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049095390000000001 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005627+0000", 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-06 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030812+0000", 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025185383 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030868+0000", 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6032000000000002e-05 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.683977892999998, 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301912289999999, 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000735+0000", 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000739+0000", 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0119999999999999e-06 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000747+0000", 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1159999999999999e-06 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000751+0000", 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048937850000000003 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5700000000000002e-07 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030914+0000", 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252686520000001 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030925+0000", 2024-09-15T03:41:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1248e-05 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.683869491999999, 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301360509999999, 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000843+0000", 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000856+0000", 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2296999999999999e-05 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000869+0000", 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3045e-05 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000878+0000", 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6749999999999997e-06 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005662+0000", 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047836379999999998 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005663+0000", 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3300000000000002e-07 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030967+0000", 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025303984 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030978+0000", 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1972e-05 2024-09-15T03:41:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.4s2 REQUEST e53)", 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.908439806000001, 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27094768699999999, 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776274+0000", 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.007e-06 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776283+0000", 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9870000000000005e-06 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047099+0000", 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.270815584 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047106+0000", 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7249999999999997e-06 2024-09-15T03:41:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047165+0000", 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8955000000000001e-05 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047186+0000", 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047220+0000", 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4156999999999998e-05 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.10s3 REQUEST e61)", 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 20.972063447, 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.318056805, 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712649+0000", 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4e-07 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712654+0000", 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5340000000000001e-06 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030357+0000", 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.31770321000000001 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030364+0000", 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.494e-06 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030652+0000", 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000288733 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030676+0000", 2024-09-15T03:41:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4046e-05 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030705+0000", 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8847999999999999e-05 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "replica_scrub(pg: 3.10s3,from:0'0,to:58'33,epoch:62/49,start:3:08000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 20.651811016, 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072588579, 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032902+0000", 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7899999999999997e-07 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032909+0000", 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4200000000000001e-06 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105400+0000", 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072491085999999996 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105407+0000", 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6800000000000004e-06 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105450+0000", 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2812999999999998e-05 2024-09-15T03:41:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105459+0000", 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4830000000000005e-06 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105489+0000", 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0218000000000001e-05 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:494 3.bs0 3:d31b706c:::smithi10536496-7:head [copy-from ver 32] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.650887416, 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.39424540800000002, 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 494 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033827+0000", 2024-09-15T03:41:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3460000000000001e-06 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033834+0000", 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5850000000000002e-06 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033850+0000", 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5528000000000001e-05 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033857+0000", 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4819999999999997e-06 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033931+0000", 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3873000000000002e-05 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.034026+0000", 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4959999999999999e-05 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.052129+0000", 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018102593 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.054670+0000", 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025406600000000001 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.428003+0000", 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.37333323699999998 2024-09-15T03:41:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.428070+0000", 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7144000000000006e-05 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPushReply(3.5s2 71/69 [PushReplyOp(3:a7103f89:::smithi10536496-39:head)])", 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.606425135, 2024-09-15T03:41:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.38581299400000002, 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.02e-07 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078289+0000", 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.311e-06 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463477+0000", 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.38518750299999999 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463488+0000", 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1687e-05 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463605+0000", 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011645 2024-09-15T03:41:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.464100+0000", 2024-09-15T03:41:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00049494100000000004 2024-09-15T03:41:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:554 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.556367803000001, 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5249715749999999, 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 554 2024-09-15T03:41:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128346+0000", 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3520000000000001e-06 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128356+0000", 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.876e-06 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128385+0000", 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9113000000000001e-05 2024-09-15T03:41:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128394+0000", 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7099999999999996e-06 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128458+0000", 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4274000000000002e-05 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128463+0000", 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6240000000000001e-06 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.649581+0000", 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.521117751 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.649784+0000", 2024-09-15T03:41:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000202759 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.653316+0000", 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035321160000000001 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:555 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.556302413000001, 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.525273152, 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 555 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9699999999999996e-07 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128414+0000", 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.912e-06 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128423+0000", 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3740000000000004e-06 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128428+0000", 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7840000000000003e-06 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128504+0000", 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5908999999999994e-05 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128505+0000", 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.204e-06 2024-09-15T03:41:49.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650194+0000", 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5216883160000001 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650271+0000", 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7257999999999998e-05 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.653683+0000", 2024-09-15T03:41:49.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003411898 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:556 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 11.556261878000001, 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5256133200000002, 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:49.792 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 556 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:49.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128451+0000", 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128455+0000", 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0350000000000003e-06 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:49.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128524+0000", 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9486000000000005e-05 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128531+0000", 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6089999999999997e-06 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128573+0000", 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1913999999999999e-05 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128574+0000", 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.155e-06 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650502+0000", 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5219284549999998 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650630+0000", 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012748800000000001 2024-09-15T03:41:49.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.654063+0000", 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034336480000000001 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:49.797 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:49.798 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:49.798 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:49.799 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:49.800 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:49.801 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:49.801 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:49.797+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:49.900 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:49.899+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:49.942 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:49.942 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:49.942 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:49.942 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:49.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:50.001 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:50.000+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:296: done (4 left) 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:298: done (3 left) 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:299: read oid 48 snap 18 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:299: expect deleted 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:300: read oid 19 snap 18 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:300: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:41:50.010 INFO:tasks.rados.rados.0.smithi105.stdout:301: snap_remove snap 24 2024-09-15T03:41:50.064 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:50.064 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:50.064 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:50.064 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:50.064 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:50.072 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:50.101 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:50.101+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:50.160 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:41:50.176 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:50.176 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:50.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:50.176 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:50.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=77, reqid=client.4562.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=47'19))", 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 36.145965048000001, 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92662449099999999, 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028438+0000", 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2099999999999996e-07 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028447+0000", 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8759999999999994e-06 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954390+0000", 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92594294600000004 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954412+0000", 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2218e-05 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954469+0000", 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7000999999999999e-05 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954479+0000", 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5489999999999995e-06 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955037+0000", 2024-09-15T03:41:50.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055842499999999998 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955062+0000", 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4754999999999999e-05 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:310 3.as0 3.bc9e946a (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e48)", 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:50.181 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.163813918000002, 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99312189699999998, 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 310 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:50.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010590+0000", 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.581e-06 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010594+0000", 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.98e-06 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010608+0000", 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4251e-05 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010617+0000", 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7819999999999999e-06 2024-09-15T03:41:50.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010695+0000", 2024-09-15T03:41:50.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8144000000000003e-05 2024-09-15T03:41:50.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:50.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010699+0000", 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8659999999999999e-06 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003696+0000", 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99299729199999998 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003710+0000", 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4001e-05 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.189 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 34.169720800999997, 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048967167, 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.190 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004683+0000", 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2899999999999999e-06 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004689+0000", 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9229999999999999e-06 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004703+0000", 2024-09-15T03:41:50.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004719+0000", 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6095000000000001e-05 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005604+0000", 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00088508100000000002 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005608+0000", 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.247e-06 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030447+0000", 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.024839515 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030545+0000", 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7453000000000004e-05 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.053649+0000", 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023104197999999999 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 34.169670209000003, 2024-09-15T03:41:50.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.049664036, 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:50.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004733+0000", 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0100000000000004e-07 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004737+0000", 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9049999999999999e-06 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004745+0000", 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7540000000000006e-06 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004752+0000", 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9000000000000006e-06 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005631+0000", 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087836100000000003 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005632+0000", 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.47e-07 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030868+0000", 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252366669999999 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030963+0000", 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4665000000000003e-05 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054396+0000", 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023433236 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:50.201 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 34.169644253999998, 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0501537240000001, 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004759+0000", 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.82e-07 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004762+0000", 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2629999999999999e-06 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004769+0000", 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0260000000000002e-06 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004780+0000", 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0569e-05 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005657+0000", 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087720500000000004 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005658+0000", 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.159e-06 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031236+0000", 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0255778339999999 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031320+0000", 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4217999999999999e-05 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054912+0000", 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023591867999999998 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:412 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~688128] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:50.207 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 23.090934736000001, 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013464116999999999, 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 412 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083474+0000", 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084200+0000", 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00072616300000000005 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084223+0000", 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2501e-05 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084230+0000", 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8709999999999997e-06 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084286+0000", 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5943000000000003e-05 2024-09-15T03:41:50.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084297+0000", 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1009000000000001e-05 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.089736+0000", 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054388459999999998 2024-09-15T03:41:50.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.089847+0000", 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000110355 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.090995+0000", 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011485 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094915+0000", 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039203249999999997 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.096868+0000", 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019529199999999999 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.096932+0000", 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3402e-05 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:413 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~540672] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 23.087761541999999, 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012499748, 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 413 2024-09-15T03:41:50.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086643+0000", 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0509999999999999e-06 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087150+0000", 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00050739100000000005 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087172+0000", 2024-09-15T03:41:50.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1903e-05 2024-09-15T03:41:50.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087182+0000", 2024-09-15T03:41:50.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:41:50.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087222+0000", 2024-09-15T03:41:50.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0246000000000001e-05 2024-09-15T03:41:50.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:50.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087227+0000", 2024-09-15T03:41:50.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0019999999999997e-06 2024-09-15T03:41:50.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.091754+0000", 2024-09-15T03:41:50.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045271299999999999 2024-09-15T03:41:50.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.091849+0000", 2024-09-15T03:41:50.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4918000000000002e-05 2024-09-15T03:41:50.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.092538+0000", 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068922000000000005 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095065+0000", 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025265919999999998 2024-09-15T03:41:50.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099059+0000", 2024-09-15T03:41:50.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039940599999999998 2024-09-15T03:41:50.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099141+0000", 2024-09-15T03:41:50.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1749000000000006e-05 2024-09-15T03:41:50.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.225 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.226 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.226 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:414 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~229376] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:50.226 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:50.226 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 23.086492285999999, 2024-09-15T03:41:50.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01143303, 2024-09-15T03:41:50.227 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.227 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.228 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 414 2024-09-15T03:41:50.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.228 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:50.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:50.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087911+0000", 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2190000000000001e-06 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088148+0000", 2024-09-15T03:41:50.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023696200000000001 2024-09-15T03:41:50.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088161+0000", 2024-09-15T03:41:50.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3091000000000001e-05 2024-09-15T03:41:50.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088167+0000", 2024-09-15T03:41:50.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.48e-06 2024-09-15T03:41:50.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088196+0000", 2024-09-15T03:41:50.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9147999999999999e-05 2024-09-15T03:41:50.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:50.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088200+0000", 2024-09-15T03:41:50.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8060000000000001e-06 2024-09-15T03:41:50.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093070+0000", 2024-09-15T03:41:50.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048699090000000004 2024-09-15T03:41:50.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093181+0000", 2024-09-15T03:41:50.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000111192 2024-09-15T03:41:50.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093854+0000", 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067265900000000004 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095066+0000", 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001212762 2024-09-15T03:41:50.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099287+0000", 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004220985 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099343+0000", 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5816999999999998e-05 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:415 3.19s0 3:9992a557:::smithi10536496-15:head [setxattr _header (57) in=7b] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 23.08622742, 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012852831, 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 415 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.240 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9399999999999995e-07 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088178+0000", 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7439999999999999e-06 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088183+0000", 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1170000000000003e-06 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088186+0000", 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1109999999999999e-06 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088229+0000", 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2979999999999998e-05 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088237+0000", 2024-09-15T03:41:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094281+0000", 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060443909999999997 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094345+0000", 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3138000000000004e-05 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094761+0000", 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041665300000000001 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095343+0000", 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058145699999999996 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.100956+0000", 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056129270000000002 2024-09-15T03:41:50.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101028+0000", 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2167000000000002e-05 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:416 3.19s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 23.086211974000001, 2024-09-15T03:41:50.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013816281999999999, 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 416 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088191+0000", 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9900000000000002e-07 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088194+0000", 2024-09-15T03:41:50.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5140000000000002e-06 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088201+0000", 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8070000000000002e-06 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088204+0000", 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2389999999999998e-06 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088260+0000", 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5937999999999999e-05 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088266+0000", 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6330000000000001e-06 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094947+0000", 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066809970000000002 2024-09-15T03:41:50.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095006+0000", 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8712000000000003e-05 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101269+0000", 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062632080000000001 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101315+0000", 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6714e-05 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.102007+0000", 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000691221 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:423 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~704512] snapc 19=[19,18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:50.252 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.069139981999999, 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011973101, 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 423 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105264+0000", 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.928e-06 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106293+0000", 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010284280000000001 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106312+0000", 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8940000000000002e-05 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106320+0000", 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3610000000000001e-06 2024-09-15T03:41:50.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106383+0000", 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106483+0000", 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9992000000000001e-05 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.107864+0000", 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013807120000000001 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109771+0000", 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019068659999999999 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.117186+0000", 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074148460000000001 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.117235+0000", 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9840999999999999e-05 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:424 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~688128] snapc 19=[19,18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.109565+0000", 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.064837233999999, 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010381939, 2024-09-15T03:41:50.258 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 424 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109565+0000", 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109565+0000", 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109567+0000", 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.889e-06 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110243+0000", 2024-09-15T03:41:50.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067574099999999999 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110257+0000", 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.401e-05 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110262+0000", 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7400000000000004e-06 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110298+0000", 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.608e-05 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110385+0000", 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6967999999999998e-05 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.112045+0000", 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016602500000000001 2024-09-15T03:41:50.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.114536+0000", 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024912340000000002 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.119850+0000", 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053136900000000003 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.119947+0000", 2024-09-15T03:41:50.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7337000000000007e-05 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:427 3.19s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.110386+0000", 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.064016091999999, 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011215087, 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.264 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 427 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110386+0000", 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110386+0000", 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110387+0000", 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0499999999999999e-07 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110390+0000", 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.022e-06 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110395+0000", 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8949999999999997e-06 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.110397+0000", 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6769999999999999e-06 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.115508+0000", 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051103499999999996 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.115605+0000", 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7818000000000004e-05 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.120670+0000", 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050643370000000004 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.120726+0000", 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6718999999999997e-05 2024-09-15T03:41:50.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.121601+0000", 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087486400000000005 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:448 3.as0 3:5dde9339:::smithi10536496-16:head [rollback 17] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.134752055, 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012127656000000001, 2024-09-15T03:41:50.270 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 448 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039651+0000", 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5000000000000003e-07 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039654+0000", 2024-09-15T03:41:50.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1209999999999998e-06 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039661+0000", 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4830000000000002e-06 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039664+0000", 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.016e-06 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041239+0000", 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015745449999999999 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041330+0000", 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1223000000000004e-05 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041870+0000", 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00053994899999999996 2024-09-15T03:41:50.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043742+0000", 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018719940000000001 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.051733+0000", 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079906860000000003 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.051778+0000", 2024-09-15T03:41:50.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5089000000000001e-05 2024-09-15T03:41:50.276 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.276 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.276 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:449 3.as0 3:5dde9339:::smithi10536496-16:head [append 0~0] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.134734358999999, 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018300219999999999, 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 449 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.89e-07 2024-09-15T03:41:50.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039671+0000", 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2910000000000002e-06 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039676+0000", 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.383e-06 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039680+0000", 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4879999999999999e-06 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042268+0000", 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025880899999999999 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042414+0000", 2024-09-15T03:41:50.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000146312 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042875+0000", 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00046076800000000002 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044451+0000", 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001575833 2024-09-15T03:41:50.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.057930+0000", 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013479326 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.057968+0000", 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.824e-05 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.282 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/51 ECSubWriteReply(tid=49, last_complete=65'44, committed=1, applied=1))", 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.132725768, 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.47081936099999999, 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0900000000000001e-07 2024-09-15T03:41:50.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041681+0000", 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8970000000000001e-06 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512389+0000", 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.470707388 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512401+0000", 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2897999999999999e-05 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512468+0000", 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6607999999999997e-05 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512496+0000", 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7861e-05 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:625 3.as0 3:57d09725:::smithi10536496-48:head [append 0~475136] snapc 28=[28,27,26,25,24] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.14598114, 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011332543, 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 625 2024-09-15T03:41:50.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028422+0000", 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2160000000000001e-06 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028856+0000", 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043326999999999998 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028866+0000", 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:50.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028871+0000", 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028939+0000", 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8388999999999999e-05 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029004+0000", 2024-09-15T03:41:50.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4908999999999998e-05 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029559+0000", 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000555465 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030619+0000", 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010596799999999999 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039713+0000", 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090945100000000001 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039754+0000", 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0303999999999999e-05 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.292 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:626 3.as0 3:57d09725:::smithi10536496-48:head [setxattr _header (57) in=7b] snapc 28=[28,27,26,25,24] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.1455236759999998, 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011104786, 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 626 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028879+0000", 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:41:50.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028881+0000", 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.069e-06 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028886+0000", 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2059999999999998e-06 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028890+0000", 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.963e-06 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029913+0000", 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010229080000000001 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030002+0000", 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8792e-05 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030403+0000", 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00040096199999999999 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030723+0000", 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00032022399999999998 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039938+0000", 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0092151850000000007 2024-09-15T03:41:50.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039983+0000", 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5018000000000002e-05 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:627 3.as0 3:57d09725:::smithi10536496-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.1454971760000001, 2024-09-15T03:41:50.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012208765, 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 627 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028906+0000", 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028907+0000", 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.544e-06 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028911+0000", 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028914+0000", 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030561+0000", 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016473379999999999 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030614+0000", 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2911000000000002e-05 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.040187+0000", 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095729660000000005 2024-09-15T03:41:50.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.040275+0000", 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7498999999999997e-05 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.041114+0000", 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000839151 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/51 ECSubWriteReply(tid=88, last_complete=77'59, committed=1, applied=1))", 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.044436+0000", 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.1299659640000002, 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.36005547199999999, 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044436+0000", 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044436+0000", 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044437+0000", 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.04e-06 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044442+0000", 2024-09-15T03:41:50.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6909999999999996e-06 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404387+0000", 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.35994481 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404393+0000", 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8089999999999998e-06 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404446+0000", 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2889000000000001e-05 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404492+0000", 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6233000000000003e-05 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.307 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:50.309 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:50.309 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:50.308+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:50.409 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:50.408+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:50.451 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:50.452 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:50.452 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:50.452 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:50.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:50.510 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:50.509+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:50.562 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:50.562 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:50.562 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:50.563 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:50.563 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:50.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:50.611 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:50.611+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:50.674 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:50.674 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 48.654441712000001, 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023454370000000001, 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018416+0000", 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5300000000000004e-07 2024-09-15T03:41:50.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018457+0000", 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021725800000000001 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018754+0000", 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0282000000000001e-05 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018806+0000", 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018858+0000", 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1873000000000003e-05 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020727+0000", 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001868897 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020761+0000", 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4165999999999998e-05 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 48.601912964999997, 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084043045999999996, 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070945+0000", 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.124e-06 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070959+0000", 2024-09-15T03:41:50.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4212999999999999e-05 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154508+0000", 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083548836000000001 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154515+0000", 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154566+0000", 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1025e-05 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154578+0000", 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154965+0000", 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038690300000000002 2024-09-15T03:41:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154987+0000", 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2172999999999999e-05 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=12, reqid=client.4562.0:150, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 48.518612705999999, 2024-09-15T03:41:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035676890000000002, 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154245+0000", 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154487+0000", 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000242308 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156728+0000", 2024-09-15T03:41:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022401589999999998 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156736+0000", 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3259999999999996e-06 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156781+0000", 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4583000000000003e-05 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156789+0000", 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8359999999999998e-06 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157761+0000", 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097187399999999996 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157812+0000", 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0695999999999997e-05 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.688 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=14, reqid=client.4562.0:152, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 48.515668816000002, 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028494564, 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157189+0000", 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3399999999999998e-07 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157206+0000", 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7207e-05 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185008+0000", 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027801691999999999 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185015+0000", 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185068+0000", 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2333000000000002e-05 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185077+0000", 2024-09-15T03:41:50.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0259999999999996e-06 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185660+0000", 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058343299999999996 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185683+0000", 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714e-05 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.693 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=15, reqid=client.4562.0:164, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 48.488189046999999, 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027286849999999998, 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184669+0000", 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000005e-07 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184986+0000", 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000317027 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015847960000000001 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186578+0000", 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3220000000000004e-06 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186619+0000", 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1186000000000002e-05 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186629+0000", 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187377+0000", 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074841899999999995 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187397+0000", 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9233999999999999e-05 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=17, reqid=client.4562.0:166, at_version=35'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 48.485972064999999, 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099130431000000005, 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186886+0000", 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186906+0000", 2024-09-15T03:41:50.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0482000000000001e-05 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285393+0000", 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098487125999999994 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285401+0000", 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3780000000000003e-06 2024-09-15T03:41:50.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285446+0000", 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028999999999999e-05 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285453+0000", 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285995+0000", 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00054192900000000002 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.286015+0000", 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0579000000000001e-05 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:50.704 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 39.651343879999999, 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030110250000000001, 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8299999999999998e-07 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022295+0000", 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078131100000000003 2024-09-15T03:41:50.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022331+0000", 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6103000000000003e-05 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022338+0000", 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6410000000000002e-06 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022426+0000", 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7366000000000003e-05 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022437+0000", 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.131e-05 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024502+0000", 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002065271 2024-09-15T03:41:50.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024524+0000", 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1739999999999999e-05 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 47/32 ECSubWrite(tid=66, reqid=osd.11.0:66, at_version=47'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 36.664325957000003, 2024-09-15T03:41:50.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003545678, 2024-09-15T03:41:50.710 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.710 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.710 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.29e-07 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008562+0000", 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0871999999999999e-05 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008606+0000", 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4057999999999997e-05 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008618+0000", 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878999999999999e-05 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008722+0000", 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104236 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008732+0000", 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0142999999999999e-05 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012049+0000", 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033168920000000001 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012077+0000", 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7368999999999998e-05 2024-09-15T03:41:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 32.648316823999998, 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042098190000000001, 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9600000000000001e-07 2024-09-15T03:41:50.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024564+0000", 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3714e-05 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024603+0000", 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9048000000000003e-05 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024610+0000", 2024-09-15T03:41:50.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1520000000000003e-06 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024717+0000", 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000106872 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024728+0000", 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028684+0000", 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039562190000000004 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028750+0000", 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5658000000000001e-05 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.18s6 51/49 ECSubWrite(tid=153, reqid=client.4562.0:369, at_version=51'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:50.720 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 32.637626627000003, 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028221240000000001, 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035231+0000", 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4699999999999998e-07 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036344+0000", 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001113721 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036395+0000", 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0247000000000002e-05 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036403+0000", 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4630000000000001e-06 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036467+0000", 2024-09-15T03:41:50.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3876999999999998e-05 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036480+0000", 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2818e-05 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.038029+0000", 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015492629999999999 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.038052+0000", 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3487999999999999e-05 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 56/32 ECSubWrite(tid=88, reqid=osd.11.0:88, at_version=56'26, trim_to=0'0, roll_forward_to=51'24))", 2024-09-15T03:41:50.725 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.609638576000002, 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003747588, 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063219+0000", 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063262+0000", 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3173000000000002e-05 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063457+0000", 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019513700000000001 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063464+0000", 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6120000000000004e-06 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063517+0000", 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3477999999999999e-05 2024-09-15T03:41:50.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063528+0000", 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.112e-05 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066940+0000", 2024-09-15T03:41:50.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034120299999999999 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066966+0000", 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5724e-05 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 62/32 ECSubWrite(tid=60, reqid=osd.2.0:60, at_version=62'32, trim_to=0'0, roll_forward_to=58'30))", 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 21.630666376000001, 2024-09-15T03:41:50.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025058089999999999, 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042217+0000", 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5256e-05 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042269+0000", 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2383999999999999e-05 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042281+0000", 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1977000000000001e-05 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042333+0000", 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2537999999999997e-05 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042345+0000", 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1307e-05 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044670+0000", 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023257120000000002 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044696+0000", 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5996e-05 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.734 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 16.617185541000001, 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025217009999999999, 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:50.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055672+0000", 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.18e-07 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056050+0000", 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00037845499999999998 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056164+0000", 2024-09-15T03:41:50.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113744 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056168+0000", 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056220+0000", 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2176999999999999e-05 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056230+0000", 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8500000000000006e-06 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058166+0000", 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001936243 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058193+0000", 2024-09-15T03:41:50.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7137000000000001e-05 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 77/71 ECSubWrite(tid=149, reqid=osd.4.0:149, at_version=77'31, trim_to=0'0, roll_forward_to=75'29))", 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.6505155309999999, 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021045639000000001, 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022342+0000", 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:41:50.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022376+0000", 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4474999999999998e-05 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042674+0000", 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020297453 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042677+0000", 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5939999999999998e-06 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042725+0000", 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7528000000000003e-05 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042736+0000", 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1420000000000001e-05 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.043360+0000", 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00062334599999999999 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.043387+0000", 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7421000000000001e-05 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.744 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(osd.5.22:4 3.5s0 3:a7103f89:::smithi10536496-39:head [assert-version v41,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.6313569189999999, 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097123769999999995, 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041501+0000", 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.218e-06 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041508+0000", 2024-09-15T03:41:50.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.421e-06 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041525+0000", 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7164e-05 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041531+0000", 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9279999999999998e-06 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041589+0000", 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8352e-05 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041750+0000", 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000160657 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.051212+0000", 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094626369999999994 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.748 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpReadReply(3.5s0 77/72 ECSubReadReply(tid=4, attrs_read=0))", 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.625653496, 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034924600000000002, 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:41:50.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047204+0000", 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4000000000000001e-07 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050148+0000", 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029437109999999999 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050597+0000", 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044960600000000002 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050605+0000", 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.695e-06 2024-09-15T03:41:50.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050651+0000", 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5874000000000002e-05 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050696+0000", 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4934e-05 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 78/71 ECSubWrite(tid=91, reqid=osd.2.0:91, at_version=78'61, trim_to=0'0, roll_forward_to=77'59))", 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.6454497249999998, 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002303907, 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:50.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0800000000000001e-07 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027432+0000", 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4953e-05 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027482+0000", 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9756999999999998e-05 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027493+0000", 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1285e-05 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027561+0000", 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7124000000000003e-05 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027571+0000", 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0756e-05 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.029686+0000", 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021143490000000002 2024-09-15T03:41:50.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.029711+0000", 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5474999999999999e-05 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 78/71 ECSubWrite(tid=67, reqid=osd.14.0:67, at_version=78'46, trim_to=0'0, roll_forward_to=77'44))", 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.6445362020000003, 2024-09-15T03:41:50.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002840736, 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5900000000000001e-07 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028364+0000", 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3287000000000001e-05 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028453+0000", 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8350999999999995e-05 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028460+0000", 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4959999999999999e-06 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028524+0000", 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3696999999999997e-05 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028536+0000", 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2666999999999999e-05 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031128+0000", 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025911660000000002 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031162+0000", 2024-09-15T03:41:50.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3912999999999999e-05 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 78/71 ECSubWrite(tid=92, reqid=osd.2.0:92, at_version=78'63, trim_to=0'0, roll_forward_to=77'59))", 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.6416197869999998, 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003315836, 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.762 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031238+0000", 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0669999999999999e-06 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031262+0000", 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4187999999999999e-05 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031302+0000", 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9521999999999999e-05 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031309+0000", 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0489999999999998e-06 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032119+0000", 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080987699999999997 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032128+0000", 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2089999999999994e-06 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.034523+0000", 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023950349999999998 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.034553+0000", 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9889000000000001e-05 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 78/71 ECSubWrite(tid=151, reqid=osd.4.0:151, at_version=78'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.6406417260000001, 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043349979999999996, 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:50.767 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:50.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.35e-07 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032236+0000", 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0922000000000001e-05 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032321+0000", 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032327+0000", 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.516e-06 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032717+0000", 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038967199999999998 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:50.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032727+0000", 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.876e-06 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036523+0000", 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037960239999999998 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036550+0000", 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7307000000000001e-05 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.771 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.772 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.772 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:50.772 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:50.772 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:50.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:50.774 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:50.772+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:50.874 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:50.873+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:50.888 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:50.888 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:50.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:50.974 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:50.974+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:51.000 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:51.000 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:51.011 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:41:51.012 INFO:tasks.rados.rados.0.smithi105.stdout:301: done (5 left) 2024-09-15T03:41:51.012 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 40 current snap is 25 2024-09-15T03:41:51.012 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing write tid 1 to smithi10536496-22 2024-09-15T03:41:51.012 INFO:tasks.rados.rados.0.smithi105.stdout:295: finishing copy_from racing read to smithi10536496-25 2024-09-15T03:41:51.012 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing write tid 2 to smithi10536496-22 2024-09-15T03:41:51.012 INFO:tasks.rados.rados.0.smithi105.stdout:302: seq_num 122 ranges {1867776=507904,2375680=425984,2801664=442368,3244032=540672,3784704=147456} 2024-09-15T03:41:51.025 INFO:tasks.rados.rados.0.smithi105.stdout:302: writing smithi10536496-40 from 1867776 to 2375680 tid 1 2024-09-15T03:41:51.027 INFO:tasks.rados.rados.0.smithi105.stdout:302: writing smithi10536496-40 from 2375680 to 2801664 tid 2 2024-09-15T03:41:51.030 INFO:tasks.rados.rados.0.smithi105.stdout:302: writing smithi10536496-40 from 2801664 to 3244032 tid 3 2024-09-15T03:41:51.033 INFO:tasks.rados.rados.0.smithi105.stdout:302: writing smithi10536496-40 from 3244032 to 3784704 tid 4 2024-09-15T03:41:51.034 INFO:tasks.rados.rados.0.smithi105.stdout:302: writing smithi10536496-40 from 3784704 to 3932160 tid 5 2024-09-15T03:41:51.034 INFO:tasks.rados.rados.0.smithi105.stdout:299: done (5 left) 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:303: delete oid 4 current snap is 25 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing write tid 3 to smithi10536496-22 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing write tid 4 to smithi10536496-22 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing write tid 5 to smithi10536496-22 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing write tid 6 to smithi10536496-22 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 34 (ObjNum 121 snap 24 seq_num 121) dirty exists 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:294: left oid 22 (ObjNum 121 snap 24 seq_num 121) 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:297: finishing copy_from racing read to smithi10536496-39 2024-09-15T03:41:51.035 INFO:tasks.rados.rados.0.smithi105.stdout:300: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-15T03:41:51.043 INFO:tasks.rados.rados.0.smithi105.stdout:295: finishing copy_from to smithi10536496-25 2024-09-15T03:41:51.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 32 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-15T03:41:51.043 INFO:tasks.rados.rados.0.smithi105.stdout:297: finishing copy_from to smithi10536496-39 2024-09-15T03:41:51.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:41:51.043 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 1 to smithi10536496-40 2024-09-15T03:41:51.043 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 2 to smithi10536496-40 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:294: done (5 left) 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:295: done (4 left) 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:297: done (3 left) 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:300: done (2 left) 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:303: done (1 left) 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:304: snap_create 2024-09-15T03:41:51.044 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 3 to smithi10536496-40 2024-09-15T03:41:51.045 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 4 to smithi10536496-40 2024-09-15T03:41:51.045 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 5 to smithi10536496-40 2024-09-15T03:41:51.045 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 6 to smithi10536496-40 2024-09-15T03:41:51.046 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing write tid 7 to smithi10536496-40 2024-09-15T03:41:51.046 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 64 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-15T03:41:51.046 INFO:tasks.rados.rados.0.smithi105.stdout:302: left oid 40 (ObjNum 122 snap 25 seq_num 122) 2024-09-15T03:41:51.075 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:51.074+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:51.141 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [3, 4] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15, 3] 2024-09-15T03:41:51.141 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:41:51.141 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-15T03:41:51.141 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.076274 2024-09-15T03:41:51.142 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0.07627377818388115 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.132850007000002, 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094904819999999997, 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.23e-07 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017537+0000", 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8838000000000003e-05 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017912+0000", 2024-09-15T03:41:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037478400000000002 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017919+0000", 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8010000000000004e-06 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017987+0000", 2024-09-15T03:41:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7138000000000007e-05 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017998+0000", 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0908e-05 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026958+0000", 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089603229999999992 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026988+0000", 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0467000000000002e-05 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:5, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.132123948999997, 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085434570000000008, 2024-09-15T03:41:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018224+0000", 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018225+0000", 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2619999999999999e-06 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018254+0000", 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8656000000000001e-05 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018297+0000", 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999999e-05 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018310+0000", 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3052e-05 2024-09-15T03:41:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018392+0000", 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1527999999999996e-05 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4590000000000004e-06 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026715+0000", 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083134909999999992 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026767+0000", 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599000000000002e-05 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.128739574999997, 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010778574000000001, 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021609+0000", 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021692+0000", 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999997e-05 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021735+0000", 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3581000000000002e-05 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021743+0000", 2024-09-15T03:41:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3119999999999994e-06 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021822+0000", 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8437999999999997e-05 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021830+0000", 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2989999999999997e-06 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032339+0000", 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010508768999999999 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032387+0000", 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7851e-05 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.128086791000001, 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010171065999999999, 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022262+0000", 2024-09-15T03:41:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.043e-06 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022318+0000", 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6208000000000001e-05 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022353+0000", 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4437000000000002e-05 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022362+0000", 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022408+0000", 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5775000000000001e-05 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022419+0000", 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1450999999999999e-05 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032351+0000", 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099319260000000006 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032432+0000", 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1032000000000006e-05 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.127419291000002, 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094003279999999995, 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022930+0000", 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022949+0000", 2024-09-15T03:41:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9916e-05 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022995+0000", 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5173000000000003e-05 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023004+0000", 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2990000000000002e-06 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023067+0000", 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023073+0000", 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032300+0000", 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092268320000000008 2024-09-15T03:41:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032329+0000", 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8779999999999999e-05 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.126683585000002, 2024-09-15T03:41:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087825990000000003, 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023665+0000", 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5199999999999995e-07 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023681+0000", 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-05 2024-09-15T03:41:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023713+0000", 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1726000000000001e-05 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023720+0000", 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6830000000000002e-06 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023765+0000", 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4988999999999999e-05 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023771+0000", 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9109999999999998e-06 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032437+0000", 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086659760000000006 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032447+0000", 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0222e-05 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:26, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.126633812999998, 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087050259999999994, 2024-09-15T03:41:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023714+0000", 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023715+0000", 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023733+0000", 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8442999999999999e-05 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023770+0000", 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6585999999999998e-05 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023784+0000", 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4440999999999999e-05 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023829+0000", 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5099000000000003e-05 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023836+0000", 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4400000000000002e-06 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032396+0000", 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085599330000000005 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032419+0000", 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3288000000000001e-05 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=6, reqid=client.4562.0:34, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.126054605999997, 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081451809999999996, 2024-09-15T03:41:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024293+0000", 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024294+0000", 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0070000000000001e-06 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024371+0000", 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6451000000000004e-05 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024415+0000", 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4292000000000002e-05 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024423+0000", 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0700000000000007e-06 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024479+0000", 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5974000000000003e-05 2024-09-15T03:41:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024488+0000", 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1109999999999997e-06 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032423+0000", 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079349969999999992 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032438+0000", 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5279000000000001e-05 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.123207241999999, 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013181325000000001, 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:41:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7000000000000004e-07 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027155+0000", 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027190+0000", 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5738000000000002e-05 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027199+0000", 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1999999999999994e-06 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027257+0000", 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8690000000000002e-05 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027265+0000", 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8860000000000004e-06 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040256+0000", 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012990323999999999 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040322+0000", 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6392999999999999e-05 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.122728537999997, 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012779128000000001, 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027620+0000", 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.32e-07 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027692+0000", 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000006e-05 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027773+0000", 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0605000000000005e-05 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027782+0000", 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027884+0000", 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010173100000000001 2024-09-15T03:41:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027894+0000", 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9210000000000004e-06 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040357+0000", 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463284999999999 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1026000000000003e-05 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.121440833999998, 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011382537, 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028908+0000", 2024-09-15T03:41:51.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0300000000000001e-06 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028922+0000", 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4329e-05 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028975+0000", 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2573e-05 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028983+0000", 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029027+0000", 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3912000000000002e-05 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029036+0000", 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9560000000000003e-06 2024-09-15T03:41:51.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040254+0000", 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011217632999999999 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5880999999999997e-05 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.121428100999999, 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011534467, 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028921+0000", 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.057e-06 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028959+0000", 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8217999999999999e-05 2024-09-15T03:41:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028993+0000", 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4425999999999999e-05 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029011+0000", 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7465e-05 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029270+0000", 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025923899999999998 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029280+0000", 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040431+0000", 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011151283 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040454+0000", 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3309e-05 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.120790040000003, 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013707001, 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029559+0000", 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029676+0000", 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117273 2024-09-15T03:41:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029720+0000", 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4026000000000002e-05 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029728+0000", 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3240000000000003e-06 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030022+0000", 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029338200000000002 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030037+0000", 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5082000000000001e-05 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043234+0000", 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013196548000000001 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043265+0000", 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1312000000000001e-05 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.119729599999999, 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097244270000000008, 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030619+0000", 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999998e-07 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030630+0000", 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030662+0000", 2024-09-15T03:41:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1813000000000002e-05 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030680+0000", 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8306999999999999e-05 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030722+0000", 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1823000000000003e-05 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030729+0000", 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040325+0000", 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095952650000000004 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040343+0000", 2024-09-15T03:41:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8079000000000001e-05 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.118738098000001, 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011724098000000001, 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:41:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031742+0000", 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013129600000000001 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031779+0000", 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7311000000000002e-05 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031787+0000", 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-09-15T03:41:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031842+0000", 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4617999999999998e-05 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031856+0000", 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.42e-05 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043313+0000", 2024-09-15T03:41:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011457292000000001 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043334+0000", 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=9, reqid=client.4562.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.118539656000003, 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011489871, 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:41:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031809+0000", 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0130000000000001e-06 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031830+0000", 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0801999999999998e-05 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031864+0000", 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3575000000000003e-05 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031869+0000", 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5029999999999997e-06 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032118+0000", 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024897299999999999 2024-09-15T03:41:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032128+0000", 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4879999999999996e-06 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043276+0000", 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01114878 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043298+0000", 2024-09-15T03:41:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1736999999999999e-05 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.117869042999999, 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010878669000000001, 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:41:51.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032507+0000", 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7140000000000001e-05 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032542+0000", 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4986e-05 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032549+0000", 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2130000000000002e-06 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034852+0000", 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023031089999999998 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034862+0000", 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0156000000000001e-05 2024-09-15T03:41:51.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043343+0000", 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084812489999999997 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043357+0000", 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4219000000000001e-05 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.115227011000002, 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008115901, 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035121+0000", 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035122+0000", 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035140+0000", 2024-09-15T03:41:51.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7915999999999999e-05 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035176+0000", 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6335999999999999e-05 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035184+0000", 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035238+0000", 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4048000000000002e-05 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035248+0000", 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043207+0000", 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079593690000000009 2024-09-15T03:41:51.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043237+0000", 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9788e-05 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=20, reqid=client.4562.0:159, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.989929404000002, 2024-09-15T03:41:51.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091519410000000002, 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160419+0000", 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7499999999999999e-07 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160707+0000", 2024-09-15T03:41:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028803500000000002 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160722+0000", 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5235000000000001e-05 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160727+0000", 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7600000000000002e-06 2024-09-15T03:41:51.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160775+0000", 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7732999999999999e-05 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160785+0000", 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0380999999999999e-05 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169551+0000", 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087657610000000004 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169570+0000", 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9261000000000001e-05 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "replica_scrub(pg: 3.2s5,from:0'0,to:35'2,epoch:76/51,start:3:40000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:43.748393+0000", 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.4019552260000001, 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82562717100000005, 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748393+0000", 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748393+0000", 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748394+0000", 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7100000000000011e-07 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748415+0000", 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-09-15T03:41:51.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573901+0000", 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82548617499999999 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573909+0000", 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7589999999999997e-06 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573979+0000", 2024-09-15T03:41:51.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0051999999999999e-05 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573992+0000", 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2366e-05 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.574020+0000", 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8140000000000002e-05 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.248 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:51.250 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:51.248+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:51.350 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:51.349+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:51.352 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:51.352 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:51.451 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:51.450+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:51.462 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:51.462 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:51.462 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:51.462 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:51.552 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:51.551+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:51.614 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.614 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:51.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:51.614 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:51.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:2 3.8s0 3:136df151:::smithi10536496-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.595680905000002, 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012099814, 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016623+0000", 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.99e-07 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016626+0000", 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4600000000000002e-06 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016633+0000", 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-06 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016636+0000", 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8439999999999998e-06 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018474+0000", 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018381859999999999 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.018568+0000", 2024-09-15T03:41:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4424000000000003e-05 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019035+0000", 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046713100000000002 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.024240+0000", 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052042240000000003 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028633+0000", 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043929340000000002 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028722+0000", 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9912000000000002e-05 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:3 3.8s0 3:136df151:::smithi10536496-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.595665052000001, 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016082863999999999, 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016639+0000", 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016640+0000", 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2729999999999999e-06 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.113e-06 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016650+0000", 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3489999999999999e-06 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019489+0000", 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028386459999999998 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019567+0000", 2024-09-15T03:41:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7417999999999996e-05 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029120+0000", 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095539030000000007 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029205+0000", 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4301000000000005e-05 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032721+0000", 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003516576 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:63 3.8s0 3:12b3a6c7:::smithi10536496-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.568570917999999, 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012980953, 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 63 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.72e-07 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8279999999999999e-06 2024-09-15T03:41:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043740+0000", 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9110000000000001e-06 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043742+0000", 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6460000000000002e-06 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045685+0000", 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001942706 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045767+0000", 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2405999999999997e-05 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054190+0000", 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084225670000000006 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054302+0000", 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112184 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.056714+0000", 2024-09-15T03:41:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024113329999999999 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:286 3.8s0 3:136df151:::smithi10536496-1:head [create,setxattr dj[s][nsgcdlaO`ntnnogS`sZdZO[V\\]llWi\\gjj (11296) in=40b,setxattr qRNaVOjbUaQsk\\XsiZbOhrsOSoqX_Tqndtkb\\_Sa (1297) in=40b,setxattr `hdQPrsh[aUo`rPcbh\\tWT\\PatkS^p[P_Pdmcgdi (840) in=40b,setxattr RXicf\\^UcqO_mXh^am]mfcdttd\\mfOiPqZcomOo` (1822) in=40b,setxattr oZlXqWbdtOrc_U_m[apaV_Nr[hOOUgj`sVNpq^Rp (1473) in=40b,setxattr TrdTbcloOaiWXhjih\\mdXfYZcg\\^aOrnVOj^sf`s (10104) in=40b,setxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt (1263) in=40b,setxattr ^qaVseQnk[`qOXVa`m]]NtQ\\psXaNYand^tQYrWt (1051) in=40b,setxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q (418) in=40b,setxattr cfoPTk\\o\\Qqbi^XNOW[dbpcRb`OledY]YqcmmqUr (19016) in=40b,setxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd (10104) in=35b,setxattr mtVO^NRpeT]rQZYdTadneUO_^VXn^UdhZ^\\bUQiR (769) in=40b,setxattr prahkiNnm`U[rXSatop\\lk_p\\WVkdatg[XURZrsm (4560) in=40b,setxattr isZNUepUtPtRnYQaXpfelSVVR\\apSohRfqcfV`T_ (10033) in=40b,setxattr QZkproN[Zssrnra[rrbkaSXpatcreQpt`tWbOdoO (8349) in=40b,setxattr YqVPkaeiY^Tbe\\Rr\\heXkWjtOsi`iaN[TO\\^kP^] (876) in=40b,setxattr ZdfcZ]]snWVj]dZhZ]YSjVneRRoNbk`d^een^nqN (3368) in=40b,setxattr Y^XWcNfadqaYlbNpPbVRoPtgWreNtPrhWlfbbYY] (10804) in=40b,setxattr a`l_j\\^WlXY[ekSUURUhO (10104) in=21b,setxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ (1541) in=40b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.584070152000002, 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025921639, 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 286 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:41:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028235+0000", 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.872e-06 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028332+0000", 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028352+0000", 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0262999999999999e-05 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028359+0000", 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4390000000000003e-06 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029687+0000", 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013271909999999999 2024-09-15T03:41:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029881+0000", 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000194792 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.031152+0000", 2024-09-15T03:41:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012703910000000001 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.033779+0000", 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002626724 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054105+0000", 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020326455 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054155+0000", 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0021000000000001e-05 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=0, reqid=client.4562.0:300, at_version=50'10, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.577028515999999, 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012435007, 2024-09-15T03:41:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035275+0000", 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6899999999999999e-07 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037696+0000", 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024206610000000002 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038322+0000", 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062559400000000004 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038330+0000", 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8789999999999992e-06 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038389+0000", 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8168999999999998e-05 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038400+0000", 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1083999999999999e-05 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.047676+0000", 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092765030000000002 2024-09-15T03:41:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.047710+0000", 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3847999999999998e-05 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=1, reqid=client.4562.0:301, at_version=50'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.574602755000001, 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011111194, 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037701+0000", 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0800000000000004e-07 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037845+0000", 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143296 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038740+0000", 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089520099999999998 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038747+0000", 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1459999999999999e-06 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.039546+0000", 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000799082 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.039557+0000", 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0669000000000001e-05 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048784+0000", 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092276149999999998 2024-09-15T03:41:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048812+0000", 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7477000000000001e-05 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=2, reqid=client.4562.0:302, at_version=50'12, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.574455294000003, 2024-09-15T03:41:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01098676, 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037848+0000", 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037849+0000", 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-07 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037903+0000", 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4562999999999999e-05 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038780+0000", 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087622000000000004 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038785+0000", 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1189999999999996e-06 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.044855+0000", 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060705159999999998 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.044866+0000", 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0355e-05 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048821+0000", 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039556799999999996 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048835+0000", 2024-09-15T03:41:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3805000000000001e-05 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 50/49 ECSubWrite(tid=3, reqid=client.4562.0:303, at_version=50'13, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.574396962999998, 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011244417, 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037907+0000", 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.037921+0000", 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3678e-05 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038795+0000", 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000874331 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.038799+0000", 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3930000000000001e-06 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045279+0000", 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064799879999999999 2024-09-15T03:41:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045289+0000", 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3510000000000008e-06 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.048867+0000", 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003578178 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.049151+0000", 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028403800000000002 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:357 3.8s0 3:115f2b89:::smithi10536496-36:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.450404227, 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012091859999999999, 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 357 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161899+0000", 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161901+0000", 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5510000000000001e-06 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161907+0000", 2024-09-15T03:41:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161921+0000", 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.393e-05 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161926+0000", 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1309999999999997e-06 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.161969+0000", 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2123000000000003e-05 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.162082+0000", 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011328999999999999 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.170186+0000", 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081040480000000008 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.171217+0000", 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010315389999999999 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.173948+0000", 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027302070000000001 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.173991+0000", 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3572000000000002e-05 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.cs6 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.587376216000003, 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011226435999999999, 2024-09-15T03:41:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024927+0000", 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024928+0000", 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3099999999999996e-07 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.024961+0000", 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2818000000000003e-05 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025047+0000", 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6197000000000002e-05 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025054+0000", 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3029999999999996e-06 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025108+0000", 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4744000000000003e-05 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025120+0000", 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1212999999999999e-05 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.036125+0000", 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011005910000000001 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.036154+0000", 2024-09-15T03:41:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.832e-05 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 51/49 ECSubWrite(tid=10, reqid=osd.10.0:10, at_version=51'19, trim_to=0'0, roll_forward_to=50'17))", 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.586997852000003, 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013366444, 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025307+0000", 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025333+0000", 2024-09-15T03:41:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6886999999999999e-05 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025376+0000", 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2821000000000002e-05 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025384+0000", 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025562+0000", 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017756799999999999 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025571+0000", 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-06 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038645+0000", 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013073817999999999 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038672+0000", 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7131000000000002e-05 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:438 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~753664] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.578596058999999, 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025886468999999999, 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 438 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033710+0000", 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4380000000000002e-06 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034822+0000", 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011121359999999999 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034849+0000", 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7406000000000002e-05 2024-09-15T03:41:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034859+0000", 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6830000000000003e-06 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034943+0000", 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4303999999999998e-05 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034953+0000", 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5019999999999998e-06 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039409+0000", 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044563520000000002 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039523+0000", 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113514 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.041482+0000", 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019589410000000001 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048318+0000", 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006836033 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059547+0000", 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011228802 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059594+0000", 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7358000000000003e-05 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:439 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~720896] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.572324514999998, 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019746224, 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 439 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039981+0000", 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829999999999999e-06 2024-09-15T03:41:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042101+0000", 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021201850000000001 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042116+0000", 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4994000000000001e-05 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042124+0000", 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7729999999999999e-06 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043391+0000", 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012674959999999999 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043512+0000", 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120469 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.044649+0000", 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001137327 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048320+0000", 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036706320000000001 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059680+0000", 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011360093 2024-09-15T03:41:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059725+0000", 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5472000000000001e-05 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:440 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~49152] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.570133454, 2024-09-15T03:41:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02155733, 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 440 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042171+0000", 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.39e-06 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042243+0000", 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1474e-05 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042256+0000", 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3016999999999999e-05 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042262+0000", 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4269999999999996e-06 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045435+0000", 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031722920000000002 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045540+0000", 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105648 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046184+0000", 2024-09-15T03:41:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064343199999999999 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049337+0000", 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003153089 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063687+0000", 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01435036 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063727+0000", 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0200999999999997e-05 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:441 3.8s0 3:115f2b89:::smithi10536496-36:head [setxattr _header (57) in=7b] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.570033022, 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021588698, 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 441 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5499999999999997e-07 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042275+0000", 2024-09-15T03:41:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7900000000000001e-06 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042284+0000", 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8440000000000004e-06 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042294+0000", 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6569999999999992e-06 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046742+0000", 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044484989999999999 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046840+0000", 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7651000000000004e-05 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047278+0000", 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043781499999999998 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049368+0000", 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020900620000000002 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063805+0000", 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014437324 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063859+0000", 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4301000000000001e-05 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:442 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.570004657999998, 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028656429000000001, 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 442 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042302+0000", 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4990000000000001e-06 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042309+0000", 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7139999999999996e-06 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042313+0000", 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2660000000000003e-06 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047492+0000", 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051795030000000002 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047573+0000", 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0895000000000004e-05 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064140+0000", 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016566658000000001 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064227+0000", 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199000000000003e-05 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.070955+0000", 2024-09-15T03:41:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067281440000000001 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:631 3.8s0 3:115f2b89:::smithi10536496-36:head [copy-from ver 41] snapc 28=[28,27,26,25] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.5761232700000001, 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.46301878800000001, 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 631 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036182+0000", 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9750000000000001e-06 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036188+0000", 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8980000000000001e-06 2024-09-15T03:41:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036204+0000", 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5906e-05 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036211+0000", 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6490000000000002e-06 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036287+0000", 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5835999999999997e-05 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036393+0000", 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106378 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.057720+0000", 2024-09-15T03:41:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021326686000000001 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.060505+0000", 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027857680000000001 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.499151+0000", 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.43864593699999999 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.499199+0000", 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7754999999999999e-05 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:632 3.8s0 3:115f2b89:::smithi10536496-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.5760771980000001, 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023198263, 2024-09-15T03:41:51.713 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 632 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036227+0000", 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7199999999999998e-07 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036230+0000", 2024-09-15T03:41:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7949999999999999e-06 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036238+0000", 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1829999999999994e-06 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036242+0000", 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740000000000004e-06 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036579+0000", 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033680299999999999 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for blocked object", 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036635+0000", 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6545999999999997e-05 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059280+0000", 2024-09-15T03:41:51.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022644272 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059341+0000", 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1425000000000002e-05 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059425+0000", 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3592999999999997e-05 2024-09-15T03:41:51.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 83/82 ECSubWrite(tid=75, reqid=client.4562.0:702, at_version=83'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:50.023157+0000", 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.589146816, 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012599309, 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023157+0000", 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023157+0000", 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023158+0000", 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0860000000000001e-06 2024-09-15T03:41:51.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023439+0000", 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028167600000000002 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023493+0000", 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3579999999999999e-05 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023501+0000", 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3780000000000001e-06 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023635+0000", 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013400700000000001 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.023646+0000", 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0638e-05 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.035718+0000", 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012071587999999999 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.035756+0000", 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8355999999999997e-05 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s4 83/82 ECSubWrite(tid=76, reqid=client.4562.0:703, at_version=83'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:50.025653+0000", 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5866500530000001, 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012922463, 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.025653+0000", 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.025653+0000", 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.025654+0000", 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.026062+0000", 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040711099999999999 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.026110+0000", 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8480999999999999e-05 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.026118+0000", 2024-09-15T03:41:51.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5660000000000001e-06 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.026162+0000", 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4289999999999998e-05 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.026172+0000", 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268000000000001e-05 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:51.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.038549+0000", 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012376491 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:50.038576+0000", 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7265000000000001e-05 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:51.728 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:51.730 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:51.729+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:51.802 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:51.831 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:51.830+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:51.873 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:41:51.873 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:51.933 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:51.933 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:51.931+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:51.933 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:51.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:51.953 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:51.973 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:51.974 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:51.984 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:51.984 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:41:51.984 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:41:51.984 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:41:51.984 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:51.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:51.994 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:51.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.6 primary-affinity to 0.0762738 (849982) 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:302: done (1 left) 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:304: done (0 left) 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:305: read oid 9 snap -1 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:305: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:306: read oid 28 snap 25 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:306: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:52.014 INFO:tasks.rados.rados.0.smithi105.stdout:307: snap_create 2024-09-15T03:41:52.015 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:52.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:52.019 INFO:tasks.rados.rados.0.smithi105.stdout:306: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:41:52.031 INFO:tasks.rados.rados.0.smithi105.stdout:305: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-15T03:41:52.033 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:52.032+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:52.034 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:52.034 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:52.055 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:52.075 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:52.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:52.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:52.095 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.066396615000002, 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011155541, 2024-09-15T03:41:52.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029073+0000", 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8599999999999998e-07 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029084+0000", 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0603e-05 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029486+0000", 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000402144 2024-09-15T03:41:52.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029493+0000", 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.165e-06 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029683+0000", 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000190478 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029706+0000", 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2849000000000001e-05 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040193+0000", 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010486822999999999 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040228+0000", 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4993000000000002e-05 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:41, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.065157669000001, 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098373709999999993, 2024-09-15T03:41:52.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030311+0000", 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030312+0000", 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030333+0000", 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0635000000000001e-05 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030374+0000", 2024-09-15T03:41:52.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030383+0000", 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000004e-06 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030436+0000", 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2605000000000002e-05 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030447+0000", 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1576999999999999e-05 2024-09-15T03:41:52.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040120+0000", 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096721959999999992 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040149+0000", 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9218e-05 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.063846974000001, 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013955543000000001, 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031623+0000", 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031726+0000", 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103491 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031757+0000", 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0908999999999998e-05 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031770+0000", 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2565000000000001e-05 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031863+0000", 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2622999999999997e-05 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031928+0000", 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5984999999999999e-05 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045547+0000", 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013618655 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045578+0000", 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0565999999999998e-05 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.062883802999998, 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013036834000000001, 2024-09-15T03:41:52.112 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032586+0000", 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032605+0000", 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-05 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032648+0000", 2024-09-15T03:41:52.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2763999999999999e-05 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032656+0000", 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032693+0000", 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6701e-05 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032699+0000", 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6849999999999998e-06 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045592+0000", 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012893345 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045622+0000", 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0182e-05 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=4, reqid=client.4562.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:52.117 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.060246358999997, 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010405127, 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035223+0000", 2024-09-15T03:41:52.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035224+0000", 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1800000000000005e-07 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035236+0000", 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1952e-05 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035267+0000", 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035273+0000", 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8589999999999999e-06 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035320+0000", 2024-09-15T03:41:52.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6283999999999999e-05 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.035326+0000", 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.105e-06 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045587+0000", 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010261562 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045628+0000", 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0469000000000002e-05 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=25, reqid=client.4562.0:101, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.011127453999997, 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011334317, 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.122 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084360+0000", 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7618999999999999e-05 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088093+0000", 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037333290000000001 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088098+0000", 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7010000000000004e-06 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088131+0000", 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2653000000000001e-05 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088136+0000", 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4979999999999997e-06 2024-09-15T03:41:52.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095632+0000", 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074962550000000003 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095676+0000", 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3516999999999997e-05 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=27, reqid=client.4562.0:104, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.006632773, 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098078109999999996, 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088836+0000", 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088837+0000", 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088853+0000", 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5770999999999999e-05 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095603+0000", 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067504870000000003 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095612+0000", 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4139999999999994e-06 2024-09-15T03:41:52.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095655+0000", 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3417000000000001e-05 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095665+0000", 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0006e-05 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098619+0000", 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029542119999999999 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.098644+0000", 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4975999999999999e-05 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=32, to_read={3:edf86ef6:::smithi10536496-37:head=0,4096,0}, subchunks={3:edf86ef6:::smithi10536496-37:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:52.131 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.990029088, 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039281828999999997, 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:41:52.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105441+0000", 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105447+0000", 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5010000000000001e-06 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141752+0000", 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036305044000000002 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141762+0000", 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3940000000000002e-06 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141935+0000", 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017359499999999999 2024-09-15T03:41:52.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.144722+0000", 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00278671 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=34, reqid=client.4562.0:140, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.952956061000002, 2024-09-15T03:41:52.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038895118999999999, 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142514+0000", 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.449e-05 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180934+0000", 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038406005999999999 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180939+0000", 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4270000000000001e-06 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180984+0000", 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5751000000000003e-05 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180995+0000", 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0332e-05 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181383+0000", 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038812500000000002 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181408+0000", 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5349000000000001e-05 2024-09-15T03:41:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 35/32 ECSubWrite(tid=39, reqid=client.4562.0:163, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.910468248999997, 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099943480000000001, 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185001+0000", 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185002+0000", 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.185014+0000", 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.205e-05 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194530+0000", 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095162920000000008 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194535+0000", 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4090000000000002e-06 2024-09-15T03:41:52.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194590+0000", 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4234000000000003e-05 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194595+0000", 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4369999999999998e-06 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194977+0000", 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038248600000000002 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194995+0000", 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799e-05 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=40, reqid=client.4562.0:176, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:52.144 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.900968769000002, 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229890967, 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194501+0000", 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:52.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194514+0000", 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423889+0000", 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229374775 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423894+0000", 2024-09-15T03:41:52.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0669999999999999e-06 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423938+0000", 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3967000000000001e-05 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423948+0000", 2024-09-15T03:41:52.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2920000000000001e-06 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424370+0000", 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042234599999999998 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424391+0000", 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1441999999999999e-05 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=74, reqid=client.4562.0:297, at_version=47'17, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.083187373000001, 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097576989999999999, 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.150 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012282+0000", 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.29e-07 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012308+0000", 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6588e-05 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012483+0000", 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017428699999999999 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.012490+0000", 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2690000000000002e-06 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.013059+0000", 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056855499999999999 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.013070+0000", 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1286e-05 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.022002+0000", 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089322210000000006 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.022039+0000", 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7363999999999997e-05 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=75, reqid=client.4562.0:298, at_version=47'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.081207407999997, 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01362943, 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.155 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014263+0000", 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0100000000000001e-06 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014284+0000", 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1742e-05 2024-09-15T03:41:52.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014322+0000", 2024-09-15T03:41:52.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7746000000000001e-05 2024-09-15T03:41:52.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014330+0000", 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014378+0000", 2024-09-15T03:41:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.799e-05 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014389+0000", 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027856+0000", 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013467549000000001 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027891+0000", 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4925999999999997e-05 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=76, reqid=client.4562.0:299, at_version=47'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.080132734999999, 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01257076, 2024-09-15T03:41:52.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015337+0000", 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.65e-07 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015351+0000", 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4504e-05 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015447+0000", 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5996999999999996e-05 2024-09-15T03:41:52.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015454+0000", 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4250000000000003e-06 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016973+0000", 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001519248 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016980+0000", 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4449999999999999e-06 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027896+0000", 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010915784 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027907+0000", 2024-09-15T03:41:52.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0992000000000001e-05 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:399 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.043891272, 2024-09-15T03:41:52.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014049582999999999, 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 399 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051580+0000", 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-06 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051584+0000", 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3320000000000002e-06 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051594+0000", 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.027e-05 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051600+0000", 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3500000000000002e-06 2024-09-15T03:41:52.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5226000000000003e-05 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051663+0000", 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.25e-06 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058108+0000", 2024-09-15T03:41:52.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064450269999999999 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058282+0000", 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017381800000000001 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059091+0000", 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080964600000000002 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062585+0000", 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003493875 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065538+0000", 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002952826 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065627+0000", 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339000000000006e-05 2024-09-15T03:41:52.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:400 3.15s0 3:ac8f3d24:::smithi10536496-17:head [rollback c] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.043854752000001, 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016394477000000001, 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:52.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 400 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051615+0000", 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051618+0000", 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.977e-06 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051623+0000", 2024-09-15T03:41:52.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5010000000000004e-06 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051626+0000", 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8009999999999999e-06 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051729+0000", 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010270800000000001 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051734+0000", 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059522+0000", 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007788059 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059592+0000", 2024-09-15T03:41:52.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0457999999999995e-05 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060158+0000", 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056606199999999999 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062892+0000", 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002733769 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:52.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.067964+0000", 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050720879999999998 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068009+0000", 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4685999999999999e-05 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:401 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:41:52.179 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.043821058999999, 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016504504, 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 401 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0100000000000001e-07 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051651+0000", 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051659+0000", 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:41:52.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051769+0000", 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011069 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:52.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051774+0000", 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6430000000000003e-06 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060449+0000", 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008674747 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060522+0000", 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3393999999999996e-05 2024-09-15T03:41:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060841+0000", 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031853799999999999 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062921+0000", 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002080765 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068093+0000", 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051714500000000002 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068153+0000", 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9673000000000003e-05 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.186 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.043303498, 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37552997599999999, 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-07 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.053528+0000", 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013622840000000001 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426011+0000", 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37248246099999999 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426019+0000", 2024-09-15T03:41:52.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3089999999999996e-06 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426108+0000", 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9092999999999996e-05 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426118+0000", 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0407e-05 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427667+0000", 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015489220000000001 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427696+0000", 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8337999999999999e-05 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:620 3.15s0 3:af19ecb4:::smithi10536496-23:28 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:52.191 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:43.014811+0000", 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.080657811, 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010014596000000001, 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 620 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014811+0000", 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014811+0000", 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014812+0000", 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2599999999999998e-07 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014817+0000", 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7720000000000002e-06 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014822+0000", 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7370000000000003e-06 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014826+0000", 2024-09-15T03:41:52.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6619999999999998e-06 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014982+0000", 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015647600000000001 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014989+0000", 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2489999999999998e-06 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018063+0000", 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030738950000000001 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018125+0000", 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2119000000000006e-05 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.024826+0000", 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067012599999999997 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:621 3.15s0 3:af19ecb4:::smithi10536496-23:28 [read 0~376832,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0806134380000003, 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010658079000000001, 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:52.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 621 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014856+0000", 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0499999999999999e-07 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014859+0000", 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9440000000000001e-06 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014867+0000", 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.824e-06 2024-09-15T03:41:52.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.014881+0000", 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4596000000000001e-05 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.015018+0000", 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013627299999999999 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.015024+0000", 2024-09-15T03:41:52.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7159999999999997e-06 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018326+0000", 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033014989999999998 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.018382+0000", 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6098999999999999e-05 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:41:52.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.025514+0000", 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071317230000000004 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:41:52.202 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:41:52.204 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:52.204 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:52.202+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:52.205 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:52.205 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:52.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:52.206 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:52.206 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:52.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:52.207 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:52.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:52.207 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:52.213 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:52.214 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:52.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:52.212+0000 7fb88dfd1640 0 -- 172.21.15.5:0/4046065316 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7fb8881964e0 msgr2=0x7fb8881988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T03:41:52.305 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:52.304+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:52.309 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:52.309 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:52.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:52.406 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:52.405+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:52.437 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:52.437 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:52.459 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:52.459 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:52.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:52.513 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:52.512+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:52.570 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:52.570 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:52.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:52.619 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:52.618+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:52.692 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:52.692 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:52.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:52.745 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:52.744+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:52.802 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:52.802 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:52.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:52.811 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:52.809+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:52.912 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:52.911+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:52.948 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:52.948 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:52.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:53.013 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:53.011+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:53.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:41:53.028 INFO:tasks.rados.rados.0.smithi105.stdout:305: done (2 left) 2024-09-15T03:41:53.028 INFO:tasks.rados.rados.0.smithi105.stdout:306: done (1 left) 2024-09-15T03:41:53.028 INFO:tasks.rados.rados.0.smithi105.stdout:307: done (0 left) 2024-09-15T03:41:53.028 INFO:tasks.rados.rados.0.smithi105.stdout:308: snap_remove snap 20 2024-09-15T03:41:53.057 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.066 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:53.113 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:53.113+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:53.165 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.165 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.173 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:53.215 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:53.213+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:53.270 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.270 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:53.315 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:53.314+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:53.375 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.375 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:53.417 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:53.416+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:53.481 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.481 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:53.518 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:53.517+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:53.587 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.587 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:41:53.619 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:53.618+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:41:53.720 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:53.720+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:53.797 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.798 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:41:53.822 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:53.821+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:53.905 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:53.905 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:53.914 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:41:53.958 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:53.957+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:54.023 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:54.022+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:54.024 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.024 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:41:54.025 INFO:tasks.rados.rados.0.smithi105.stdout:308: done (0 left) 2024-09-15T03:41:54.026 INFO:tasks.rados.rados.0.smithi105.stdout:309: rollback oid 28 current snap is 27 2024-09-15T03:41:54.026 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 28 to 23 2024-09-15T03:41:54.026 INFO:tasks.rados.rados.0.smithi105.stdout:310: rollback oid 12 current snap is 27 2024-09-15T03:41:54.026 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 12 to 18 2024-09-15T03:41:54.026 INFO:tasks.rados.rados.0.smithi105.stdout:311: snap_remove snap 25 2024-09-15T03:41:54.037 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:41:54.124 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:41:54.123+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:41:54.176 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.176 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.184 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:41:54.228 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:54.227+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:54.282 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.282 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.290 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:41:54.326 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:54.326+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:54.427 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:54.426+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:54.429 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.430 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.438 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:41:54.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:54.528 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:54.527+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:54.586 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:54.586 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:54.594 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.594 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.603 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:41:54.607 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:54.608 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:54.628 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:54.628 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:54.629 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:54.628+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:54.647 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:54.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:54.667 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:54.667 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:54.688 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:54.689 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:54.708 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:54.709 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.709 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:41:54.727 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:54.730 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:54.729+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:54.746 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:54.747 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:54.768 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:54.768 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:54.788 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:54.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:54.809 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:54.817 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.817 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.825 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:41:54.829 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:54.849 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:54.849 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:54.869 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:54.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:54.868+0000 7f89f6d9d640 0 -- 172.21.15.5:0/3690983119 >> [v2:172.21.15.105:6826/33116,v1:172.21.15.105:6827/33116] conn(0x7f89f81983f0 msgr2=0x7f89f819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:41:54.870 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:54.869+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:54.923 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:54.923 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:54.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:41:54.932 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:54.930+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:55.031 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:55.030+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:311: done (2 left) 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 21 current snap is 27 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:309: finishing rollback tid 0 to smithi10536496-28 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:309: finishing rollback tid 1 to smithi10536496-28 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:309: finishing rollback tid 2 to smithi10536496-28 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 67 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:310: finishing rollback tid 1 to smithi10536496-12 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:310: finishing rollback tid 2 to smithi10536496-12 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 58 (ObjNum 94 snap 14 seq_num 94) dirty exists 2024-09-15T03:41:55.037 INFO:tasks.rados.rados.0.smithi105.stdout:312: seq_num 124 ranges {344064=540672,884736=720896,1605632=786432,2392064=49152} 2024-09-15T03:41:55.042 INFO:tasks.rados.rados.0.smithi105.stdout:312: writing smithi10536496-21 from 344064 to 884736 tid 1 2024-09-15T03:41:55.046 INFO:tasks.rados.rados.0.smithi105.stdout:312: writing smithi10536496-21 from 884736 to 1605632 tid 2 2024-09-15T03:41:55.050 INFO:tasks.rados.rados.0.smithi105.stdout:312: writing smithi10536496-21 from 1605632 to 2392064 tid 3 2024-09-15T03:41:55.051 INFO:tasks.rados.rados.0.smithi105.stdout:312: writing smithi10536496-21 from 2392064 to 2441216 tid 4 2024-09-15T03:41:55.051 INFO:tasks.rados.rados.0.smithi105.stdout:309: done (2 left) 2024-09-15T03:41:55.051 INFO:tasks.rados.rados.0.smithi105.stdout:310: done (1 left) 2024-09-15T03:41:55.051 INFO:tasks.rados.rados.0.smithi105.stdout:313: snap_remove snap 26 2024-09-15T03:41:55.075 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:55.075 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:55.084 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:41:55.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:55.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:55.132 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:55.131+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:55.184 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:55.184 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:55.184 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:41:55.185 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:55.192 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:41:55.235 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:55.233+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:55.332 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:55.332 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:41:55.332 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:41:55.332 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:41:55.333 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:55.334 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:55.333+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:55.341 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:41:55.435 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:55.434+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:55.438 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:41:55.438 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:41:55.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:41:55.438 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-09-15T03:41:55.438 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:41:55.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:41:55.535 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:55.535+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:55.549 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:55.549 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:55.549 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:55.549 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:55.558 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:41:55.637 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:55.635+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:55.660 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:55.660 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:55.660 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:55.661 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:55.661 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:55.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:41:55.737 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:55.736+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:55.772 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:55.772 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:55.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:55.772 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:55.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.772 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:7 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.753406318000003, 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01304598, 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 7 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017042+0000", 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017045+0000", 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5040000000000001e-06 2024-09-15T03:41:55.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017174+0000", 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012955 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017203+0000", 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8303e-05 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017222+0000", 2024-09-15T03:41:55.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9680000000000001e-05 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021260800000000001 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017608+0000", 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017282699999999999 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018409+0000", 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080141399999999997 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020887+0000", 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002477244 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030008+0000", 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0091209519999999999 2024-09-15T03:41:55.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030088+0000", 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0897999999999997e-05 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:8 3.6s0 3:62396a93:::smithi10536496-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.753192665, 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013086021999999999, 2024-09-15T03:41:55.778 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 8 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017256+0000", 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017258+0000", 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6300000000000001e-06 2024-09-15T03:41:55.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017267+0000", 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4830000000000005e-06 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017303+0000", 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5326e-05 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017316+0000", 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3073e-05 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019094+0000", 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001778352 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019184+0000", 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0492e-05 2024-09-15T03:41:55.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.019834+0000", 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064979700000000003 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021641+0000", 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018066779999999999 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030253+0000", 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086118240000000006 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030342+0000", 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9367e-05 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:9 3.6s0 3:62396a93:::smithi10536496-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:41:55.784 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.753059034000003, 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015701551000000001, 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 9 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:55.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017390+0000", 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017391+0000", 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.23e-07 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017394+0000", 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5999999999999998e-06 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017406+0000", 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1980999999999999e-05 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.017435+0000", 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8459e-05 2024-09-15T03:41:55.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020095+0000", 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026602399999999999 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020181+0000", 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6468e-05 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030742+0000", 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010561054 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.030896+0000", 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015357899999999999 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.033091+0000", 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021953469999999998 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs1 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.748917607999999, 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050576110000000001, 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:55.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021531+0000", 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021532+0000", 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.021586+0000", 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4029000000000001e-05 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023870+0000", 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022843490000000002 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.023879+0000", 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7509999999999997e-06 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024332+0000", 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00045252999999999998 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.024353+0000", 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1382999999999998e-05 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026560+0000", 2024-09-15T03:41:55.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002206679 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.026589+0000", 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9149000000000002e-05 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:79 3.6s0 3:60903cd9:::smithi10536496-27:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:55.794 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.711269852000001, 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056797210000000004, 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 79 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059179+0000", 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059181+0000", 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.905e-06 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059725+0000", 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000544675 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059743+0000", 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7068999999999999e-05 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059749+0000", 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8680000000000001e-06 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059823+0000", 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3856999999999994e-05 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059937+0000", 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113934 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.060672+0000", 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00073485 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062007+0000", 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001335433 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064787+0000", 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027791140000000001 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064859+0000", 2024-09-15T03:41:55.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2015999999999996e-05 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:80 3.6s0 3:60903cd9:::smithi10536496-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.710667536999999, 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052634420000000001, 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 80 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059781+0000", 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059782+0000", 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3300000000000001e-07 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059787+0000", 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9010000000000002e-06 2024-09-15T03:41:55.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059797+0000", 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0475999999999999e-05 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059802+0000", 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7310000000000001e-06 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061262+0000", 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014593609999999999 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061347+0000", 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5358999999999994e-05 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.061756+0000", 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00040932300000000001 2024-09-15T03:41:55.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062312+0000", 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055575999999999998 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.064973+0000", 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026609950000000002 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065045+0000", 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1703000000000001e-05 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:81 3.6s0 3:60903cd9:::smithi10536496-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 53.710623273000003, 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055321019999999997, 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.806 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 81 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059825+0000", 2024-09-15T03:41:55.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059826+0000", 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4899999999999995e-07 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059830+0000", 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5769999999999998e-06 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059837+0000", 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4490000000000002e-06 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.059841+0000", 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.765e-06 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062037+0000", 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002196506 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.062113+0000", 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5740000000000003e-05 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065347+0000", 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003233648 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.065358+0000", 2024-09-15T03:41:55.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0868e-05 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=60, reqid=client.4562.0:281, at_version=45'25, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 43.748684613000002, 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068718160000000002, 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021764+0000", 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4499999999999999e-07 2024-09-15T03:41:55.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.021848+0000", 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4214000000000004e-05 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024178+0000", 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023298640000000001 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024186+0000", 2024-09-15T03:41:55.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3379999999999999e-06 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024242+0000", 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5829999999999999e-05 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.024252+0000", 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028612+0000", 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043595680000000003 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028636+0000", 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3896999999999998e-05 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=61, reqid=client.4562.0:282, at_version=45'26, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 43.745204534999999, 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087088040000000005, 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025244+0000", 2024-09-15T03:41:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025245+0000", 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0600000000000001e-07 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.025397+0000", 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015260800000000001 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026753+0000", 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013562590000000001 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026763+0000", 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7340000000000003e-06 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026806+0000", 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3254000000000003e-05 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.026816+0000", 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0115000000000001e-05 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033305+0000", 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00648871 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.033953+0000", 2024-09-15T03:41:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064781799999999998 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=62, reqid=client.4562.0:283, at_version=45'27, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 43.742727039999998, 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006712165, 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027722+0000", 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8899999999999999e-07 2024-09-15T03:41:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027799+0000", 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6990999999999995e-05 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028647+0000", 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00084766699999999997 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028655+0000", 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6769999999999995e-06 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028695+0000", 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0281000000000003e-05 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.028705+0000", 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7140000000000005e-06 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034407+0000", 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.005702141 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034434+0000", 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7005e-05 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=63, reqid=client.4562.0:284, at_version=45'28, trim_to=0'0, roll_forward_to=43'23))", 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 43.742646137999998, 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066515380000000002, 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027803+0000", 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.027818+0000", 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4961e-05 2024-09-15T03:41:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029147+0000", 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001328842 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029156+0000", 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8710000000000003e-06 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029196+0000", 2024-09-15T03:41:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9928999999999997e-05 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.029204+0000", 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5809999999999997e-06 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034438+0000", 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052331399999999998 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.034454+0000", 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6558e-05 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(osd.14.22:2 3.6s0 3:62396a93:::smithi10536496-3:head [assert-version v8,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 43.726943253999998, 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010013731999999999, 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:55.829 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043505+0000", 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043507+0000", 2024-09-15T03:41:55.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9290000000000001e-06 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043518+0000", 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0934000000000001e-05 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043540+0000", 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1418000000000001e-05 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043548+0000", 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.602e-06 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043611+0000", 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3034000000000007e-05 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043784+0000", 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017247700000000001 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.053519+0000", 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097353379999999996 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:549 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~671744] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.121786+0000", 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 17.648663016, 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097161899999999995, 2024-09-15T03:41:55.833 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 549 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.121786+0000", 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.121786+0000", 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.121788+0000", 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.272e-06 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122452+0000", 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000664056 2024-09-15T03:41:55.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122475+0000", 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3404000000000002e-05 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122483+0000", 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2710000000000003e-06 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122539+0000", 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6713999999999999e-05 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.122615+0000", 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5593999999999995e-05 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.123438+0000", 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082271199999999997 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.124658+0000", 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001219873 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131436+0000", 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067785520000000002 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131502+0000", 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5741999999999995e-05 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:550 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~753664] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.125979+0000", 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 17.644469539999999, 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011372897, 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.839 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 550 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.125979+0000", 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.125979+0000", 2024-09-15T03:41:55.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.125981+0000", 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.674e-06 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126650+0000", 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00066883899999999996 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126664+0000", 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4481000000000001e-05 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126671+0000", 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4359999999999999e-06 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126708+0000", 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7712000000000001e-05 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.126782+0000", 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4101000000000001e-05 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.127462+0000", 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067922600000000001 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128515+0000", 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001053567 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137298+0000", 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087826050000000006 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137352+0000", 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4255999999999997e-05 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:551 3.6s0 3:62396a93:::smithi10536496-3:head [append 0~409600] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128415+0000", 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 17.642033473000001, 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090658130000000007, 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 551 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128415+0000", 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128415+0000", 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128417+0000", 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2950000000000001e-06 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128715+0000", 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00029882200000000002 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128731+0000", 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5769000000000001e-05 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128737+0000", 2024-09-15T03:41:55.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3289999999999999e-06 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128785+0000", 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7641e-05 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128908+0000", 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012250499999999999 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.129698+0000", 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00079081699999999997 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131104+0000", 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001405534 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137443+0000", 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063393989999999999 2024-09-15T03:41:55.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.137481+0000", 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7701999999999999e-05 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:552 3.6s0 3:62396a93:::smithi10536496-3:head [setxattr _header (56) in=7b] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128748+0000", 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 17.641700577000002, 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56563410999999997, 2024-09-15T03:41:55.850 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 552 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128748+0000", 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128748+0000", 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128749+0000", 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0800000000000004e-07 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128753+0000", 2024-09-15T03:41:55.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2849999999999996e-06 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128759+0000", 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6670000000000001e-06 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128762+0000", 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7989999999999998e-06 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.130334+0000", 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015723379999999999 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.130431+0000", 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6822000000000002e-05 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.130812+0000", 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038130299999999999 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131518+0000", 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00070572400000000002 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694297+0000", 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.562779221 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694382+0000", 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5242999999999997e-05 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.855 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:553 3.6s0 3:62396a93:::smithi10536496-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 17.641683867000001, 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.566828946, 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:55.856 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 553 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128765+0000", 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128767+0000", 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.511e-06 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128770+0000", 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5700000000000001e-06 2024-09-15T03:41:55.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128772+0000", 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.181e-06 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131204+0000", 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024313859999999998 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.131295+0000", 2024-09-15T03:41:55.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0939000000000005e-05 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694733+0000", 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56343809300000003 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694791+0000", 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8378999999999998e-05 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.695594+0000", 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080257799999999997 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 71/44 ECSubWriteReply(tid=29, last_complete=71'16, committed=1, applied=1))", 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 17.636425209999999, 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56068105000000001, 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.861 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134024+0000", 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.13e-07 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.134026+0000", 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3810000000000002e-06 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694158+0000", 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.56013147900000004 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694162+0000", 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5059999999999997e-06 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694237+0000", 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4108000000000003e-05 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.694705+0000", 2024-09-15T03:41:55.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00046806300000000001 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.13s4 78/72 ECSubWrite(tid=147, reqid=client.4562.0:642, at_version=78'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 10.739087517, 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049401810000000001, 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031361+0000", 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.55e-07 2024-09-15T03:41:55.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031444+0000", 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2619000000000003e-05 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031592+0000", 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014789800000000001 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031601+0000", 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031667+0000", 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5809000000000006e-05 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031677+0000", 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9690000000000006e-06 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036275+0000", 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045977190000000001 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036301+0000", 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6818e-05 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.869 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.fs6 78/32 ECSubWrite(tid=149, reqid=osd.11.0:149, at_version=78'44, trim_to=0'0, roll_forward_to=77'40))", 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 10.738975556, 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055448110000000002, 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:55.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031473+0000", 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9299999999999999e-07 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031492+0000", 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8856e-05 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032145+0000", 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00065225899999999998 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032159+0000", 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4798e-05 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.033418+0000", 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00125884 2024-09-15T03:41:55.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.033428+0000", 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9750000000000002e-06 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036990+0000", 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003561397 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:55.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.037018+0000", 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8393000000000001e-05 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:55.874 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:55.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:41:55.876 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:55.874+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:55.976 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:41:55.975+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:41:56.017 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:56.017 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:56.017 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:41:56.045 INFO:tasks.rados.rados.0.smithi105.stdout:313: done (1 left) 2024-09-15T03:41:56.046 INFO:tasks.rados.rados.0.smithi105.stdout:314: read oid 15 snap -1 2024-09-15T03:41:56.046 INFO:tasks.rados.rados.0.smithi105.stdout:314: expect (ObjNum 87 snap 14 seq_num 87) 2024-09-15T03:41:56.046 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 10 current snap is 27 2024-09-15T03:41:56.046 INFO:tasks.rados.rados.0.smithi105.stdout:315: seq_num 125 ranges {3407872=557056,3964928=475136,4440064=507904,4947968=475136,5423104=98304} 2024-09-15T03:41:56.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:41:56.067 INFO:tasks.rados.rados.0.smithi105.stdout:315: writing smithi10536496-10 from 3407872 to 3964928 tid 1 2024-09-15T03:41:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:315: writing smithi10536496-10 from 3964928 to 4440064 tid 2 2024-09-15T03:41:56.073 INFO:tasks.rados.rados.0.smithi105.stdout:315: writing smithi10536496-10 from 4440064 to 4947968 tid 3 2024-09-15T03:41:56.076 INFO:tasks.rados.rados.0.smithi105.stdout:315: writing smithi10536496-10 from 4947968 to 5423104 tid 4 2024-09-15T03:41:56.076 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:56.075+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:56.077 INFO:tasks.rados.rados.0.smithi105.stdout:315: writing smithi10536496-10 from 5423104 to 5521408 tid 5 2024-09-15T03:41:56.077 INFO:tasks.rados.rados.0.smithi105.stdout:316: snap_create 2024-09-15T03:41:56.077 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing write tid 1 to smithi10536496-21 2024-09-15T03:41:56.077 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing write tid 2 to smithi10536496-21 2024-09-15T03:41:56.077 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing write tid 3 to smithi10536496-21 2024-09-15T03:41:56.078 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing write tid 4 to smithi10536496-21 2024-09-15T03:41:56.078 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing write tid 5 to smithi10536496-21 2024-09-15T03:41:56.078 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing write tid 6 to smithi10536496-21 2024-09-15T03:41:56.078 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 64 (ObjNum 124 snap 27 seq_num 124) dirty exists 2024-09-15T03:41:56.078 INFO:tasks.rados.rados.0.smithi105.stdout:312: left oid 21 (ObjNum 124 snap 27 seq_num 124) 2024-09-15T03:41:56.078 INFO:tasks.rados.rados.0.smithi105.stdout:314: expect (ObjNum 87 snap 14 seq_num 87) 2024-09-15T03:41:56.136 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.137 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:56.137 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:56.137 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:56.137 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:41:56.178 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:56.176+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:56.188 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 1 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 2 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 3 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 4 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 5 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 6 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing write tid 7 to smithi10536496-10 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 73 (ObjNum 125 snap 27 seq_num 125) dirty exists 2024-09-15T03:41:56.189 INFO:tasks.rados.rados.0.smithi105.stdout:315: left oid 10 (ObjNum 125 snap 27 seq_num 125) 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:101 3.bs0 3:db73af7f:::smithi10536496-34:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 54.168840144999997, 2024-09-15T03:41:56.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060658139999999999, 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 101 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082524+0000", 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082525+0000", 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-07 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082528+0000", 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0989999999999999e-06 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082540+0000", 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2673999999999999e-05 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082544+0000", 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9559999999999999e-06 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083692+0000", 2024-09-15T03:41:56.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011471949999999999 2024-09-15T03:41:56.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.083836+0000", 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014488899999999999 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084281+0000", 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044464500000000002 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085132+0000", 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00085125900000000004 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143106+0000", 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057973867999999998 2024-09-15T03:41:56.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143182+0000", 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5862e-05 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:102 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:56.261 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 54.168798535000001, 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100342765, 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 102 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:56.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082566+0000", 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7299999999999995e-07 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082569+0000", 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2279999999999998e-06 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082577+0000", 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9319999999999996e-06 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.082584+0000", 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8009999999999998e-06 2024-09-15T03:41:56.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084479+0000", 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018943709999999999 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:56.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.084576+0000", 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7615000000000006e-05 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143227+0000", 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058651048999999997 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143302+0000", 2024-09-15T03:41:56.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4547000000000003e-05 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182908+0000", 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039606548999999998 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 35/32 ECSubWriteReply(tid=25, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:56.267 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 54.165706360999998, 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057539988, 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:56.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085658+0000", 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.98e-07 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.085660+0000", 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.902e-06 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142986+0000", 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057326329000000002 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142992+0000", 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143038+0000", 2024-09-15T03:41:56.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6576999999999998e-05 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.143198+0000", 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015933200000000001 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:111 3.7s0 3:edf86ef6:::smithi10536496-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:56.271 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 54.156983087, 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050567135999999999, 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 111 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094381+0000", 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094382+0000", 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:41:56.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094384+0000", 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7489999999999999e-06 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094393+0000", 2024-09-15T03:41:56.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2500000000000006e-06 2024-09-15T03:41:56.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.094397+0000", 2024-09-15T03:41:56.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8040000000000001e-06 2024-09-15T03:41:56.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096792+0000", 2024-09-15T03:41:56.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002394565 2024-09-15T03:41:56.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:56.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.096872+0000", 2024-09-15T03:41:56.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0492000000000001e-05 2024-09-15T03:41:56.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105071+0000", 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081983339999999998 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.105164+0000", 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3202999999999995e-05 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.144948+0000", 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039784183000000001 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.279 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:141 3.bs0 3:dc608269:::smithi10536496-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 54.111360054999999, 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055455817999999997, 2024-09-15T03:41:56.280 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 141 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140004+0000", 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6600000000000003e-07 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140006+0000", 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5409999999999999e-06 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140011+0000", 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.843e-06 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.140013+0000", 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.621e-06 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002515181 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:56.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.142605+0000", 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6328999999999994e-05 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182018+0000", 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039413493000000001 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.182116+0000", 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8159000000000007e-05 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.195460+0000", 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013343384999999999 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:273 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~802816] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 45.224446010999998, 2024-09-15T03:41:56.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0316972600000001, 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026918+0000", 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.026920+0000", 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5629999999999999e-06 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027741+0000", 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00082160000000000002 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027764+0000", 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2495000000000001e-05 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027772+0000", 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-09-15T03:41:56.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027837+0000", 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5062999999999997e-05 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.027841+0000", 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1620000000000001e-06 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043211+0000", 2024-09-15T03:41:56.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.015370181 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.043389+0000", 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017807500000000001 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.044733+0000", 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013438510000000001 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048847+0000", 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041134250000000004 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058556+0000", 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097088929999999997 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.058615+0000", 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9623999999999997e-05 2024-09-15T03:41:56.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:274 3.17s0 3:ec08d906:::smithi10536496-14:head [append 0~376832] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 45.221965935999997, 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.029738034, 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 274 2024-09-15T03:41:56.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029398+0000", 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029399+0000", 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6099999999999999e-07 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029738+0000", 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00033941400000000001 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029756+0000", 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524e-05 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029763+0000", 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9609999999999999e-06 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029831+0000", 2024-09-15T03:41:56.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8169000000000004e-05 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029833+0000", 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045818+0000", 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0159847550000001 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.045920+0000", 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102085 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.046687+0000", 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00076710099999999996 2024-09-15T03:41:56.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049023+0000", 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023356919999999999 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059069+0000", 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010046019999999999 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059136+0000", 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7156999999999994e-05 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:275 3.17s0 3:ec08d906:::smithi10536496-14:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 45.221574052000001, 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0294973439999999, 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 275 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029790+0000", 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029791+0000", 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029807+0000", 2024-09-15T03:41:56.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6107e-05 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029828+0000", 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1475999999999999e-05 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029838+0000", 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9879999999999999e-06 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029885+0000", 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6887e-05 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029887+0000", 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5030000000000001e-06 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047307+0000", 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0174202429999999 2024-09-15T03:41:56.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.047434+0000", 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012744699999999999 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048033+0000", 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00059865199999999997 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.049643+0000", 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016096159999999999 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059241+0000", 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095980579999999996 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059287+0000", 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6726e-05 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:276 3.17s0 3:ec08d906:::smithi10536496-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 45.221507490999997, 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0302367509999999, 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.308 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 276 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029857+0000", 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029862+0000", 2024-09-15T03:41:56.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4259999999999996e-06 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029875+0000", 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3132999999999999e-05 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029880+0000", 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029913+0000", 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3079e-05 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.029914+0000", 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.164e-06 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048304+0000", 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.018390516 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.048405+0000", 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010111300000000001 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059337+0000", 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010931523 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.059408+0000", 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0962000000000003e-05 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:12.060093+0000", 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068532499999999997 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 42.250685568999998, 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.030189832, 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000678+0000", 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000680+0000", 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000683+0000", 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6610000000000002e-06 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000708+0000", 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5165999999999999e-05 2024-09-15T03:41:56.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000714+0000", 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6250000000000004e-06 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005624+0000", 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049095390000000001 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005627+0000", 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3799999999999998e-06 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030812+0000", 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025185383 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030868+0000", 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6032000000000002e-05 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.319 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 42.250630000000001, 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301912289999999, 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.320 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000734+0000", 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000735+0000", 2024-09-15T03:41:56.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8899999999999999e-07 2024-09-15T03:41:56.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000739+0000", 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0119999999999999e-06 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000747+0000", 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1159999999999999e-06 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000751+0000", 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1699999999999999e-06 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048937850000000003 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005645+0000", 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5700000000000002e-07 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030914+0000", 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252686520000001 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030925+0000", 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1248e-05 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s1 3.444c1059 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 42.250521599000002, 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0301360509999999, 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.328 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000842+0000", 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000843+0000", 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000856+0000", 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2296999999999999e-05 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000869+0000", 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3045e-05 2024-09-15T03:41:56.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.000878+0000", 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6749999999999997e-06 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005662+0000", 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047836379999999998 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.005663+0000", 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3300000000000002e-07 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030967+0000", 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.025303984 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.030978+0000", 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1972e-05 2024-09-15T03:41:56.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.4s2 REQUEST e53)", 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.475091913, 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27094768699999999, 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776272+0000", 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776274+0000", 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.007e-06 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:20.776283+0000", 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9870000000000005e-06 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047099+0000", 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.270815584 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047106+0000", 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7249999999999997e-06 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047165+0000", 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8955000000000001e-05 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047186+0000", 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0271999999999998e-05 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:21.047220+0000", 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4156999999999998e-05 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDScrubReserve(3.10s3 REQUEST e61)", 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.538715553999999, 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.318056805, 2024-09-15T03:41:56.337 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712648+0000", 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712649+0000", 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4e-07 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.712654+0000", 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5340000000000001e-06 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030357+0000", 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.31770321000000001 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030364+0000", 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.494e-06 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030652+0000", 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000288733 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030676+0000", 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4046e-05 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.030705+0000", 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8847999999999999e-05 2024-09-15T03:41:56.341 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "replica_scrub(pg: 3.10s3,from:0'0,to:58'33,epoch:62/49,start:3:08000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.218463122999999, 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072588579, 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032901+0000", 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032902+0000", 2024-09-15T03:41:56.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7899999999999997e-07 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.032909+0000", 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4200000000000001e-06 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105400+0000", 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072491085999999996 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.344 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105407+0000", 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6800000000000004e-06 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105450+0000", 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2812999999999998e-05 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105459+0000", 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4830000000000005e-06 2024-09-15T03:41:56.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.105489+0000", 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0218000000000001e-05 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:494 3.bs0 3:d31b706c:::smithi10536496-7:head [copy-from ver 32] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.217539522999999, 2024-09-15T03:41:56.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.39424540800000002, 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 494 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033825+0000", 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033827+0000", 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3460000000000001e-06 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033834+0000", 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5850000000000002e-06 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033850+0000", 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5528000000000001e-05 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033857+0000", 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4819999999999997e-06 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.033931+0000", 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3873000000000002e-05 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.034026+0000", 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4959999999999999e-05 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.052129+0000", 2024-09-15T03:41:56.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018102593 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.054670+0000", 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025406600000000001 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.428003+0000", 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.37333323699999998 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.428070+0000", 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7144000000000006e-05 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPushReply(3.5s2 71/69 [PushReplyOp(3:a7103f89:::smithi10536496-39:head)])", 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.173077242000002, 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.38581299400000002, 2024-09-15T03:41:56.352 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078287+0000", 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.02e-07 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.078289+0000", 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.311e-06 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463477+0000", 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.38518750299999999 2024-09-15T03:41:56.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463488+0000", 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1687e-05 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.463605+0000", 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011645 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.355 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.464100+0000", 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00049494100000000004 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:554 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.12301991, 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5249715749999999, 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.356 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 554 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128344+0000", 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128346+0000", 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3520000000000001e-06 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128356+0000", 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.876e-06 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128385+0000", 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9113000000000001e-05 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128394+0000", 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7099999999999996e-06 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128458+0000", 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4274000000000002e-05 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128463+0000", 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6240000000000001e-06 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.649581+0000", 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.521117751 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.649784+0000", 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000202759 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.653316+0000", 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035321160000000001 2024-09-15T03:41:56.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:555 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.12295452, 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.525273152, 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.362 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 555 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128410+0000", 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9699999999999996e-07 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128414+0000", 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.912e-06 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128423+0000", 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3740000000000004e-06 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128428+0000", 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7840000000000003e-06 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128504+0000", 2024-09-15T03:41:56.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5908999999999994e-05 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128505+0000", 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.204e-06 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650194+0000", 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5216883160000001 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650271+0000", 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7257999999999998e-05 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.653683+0000", 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003411898 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.367 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:556 3.bs0 3:db73af7f:::smithi10536496-34:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.122913985, 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5256133200000002, 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 556 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128450+0000", 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128451+0000", 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-09-15T03:41:56.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128455+0000", 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0350000000000003e-06 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128524+0000", 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9486000000000005e-05 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128531+0000", 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6089999999999997e-06 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128573+0000", 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1913999999999999e-05 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:38.128574+0000", 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.155e-06 2024-09-15T03:41:56.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650502+0000", 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5219284549999998 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.650630+0000", 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012748800000000001 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.372 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:39.654063+0000", 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034336480000000001 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.373 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:41:56.375 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:56.373+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:56.475 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:56.475+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:56.476 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.476 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:56.477 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:56.477 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:41:56.576 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:56.575+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:56.587 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:56.588 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.596 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:41:56.678 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:56.677+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/44 ECSubWrite(tid=77, reqid=client.4562.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=47'19))", 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 42.668677268000003, 2024-09-15T03:41:56.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92662449099999999, 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028437+0000", 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028438+0000", 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2099999999999996e-07 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.028447+0000", 2024-09-15T03:41:56.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8759999999999994e-06 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954390+0000", 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.92594294600000004 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954412+0000", 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2218e-05 2024-09-15T03:41:56.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954469+0000", 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7000999999999999e-05 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.954479+0000", 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5489999999999995e-06 2024-09-15T03:41:56.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955037+0000", 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055842499999999998 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.704 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.955062+0000", 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4754999999999999e-05 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:310 3.as0 3.bc9e946a (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e48)", 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.686526137999998, 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99312189699999998, 2024-09-15T03:41:56.705 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 310 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010588+0000", 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010590+0000", 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.581e-06 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010594+0000", 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.98e-06 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010608+0000", 2024-09-15T03:41:56.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4251e-05 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010617+0000", 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7819999999999999e-06 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010695+0000", 2024-09-15T03:41:56.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8144000000000003e-05 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:15.010699+0000", 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8659999999999999e-06 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003696+0000", 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.99299729199999998 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.003710+0000", 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4001e-05 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:294 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.692433020999999, 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.048967167, 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.712 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 294 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.714 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004682+0000", 2024-09-15T03:41:56.714 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004683+0000", 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2899999999999999e-06 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004689+0000", 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9229999999999999e-06 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004703+0000", 2024-09-15T03:41:56.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4365000000000001e-05 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004719+0000", 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6095000000000001e-05 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005604+0000", 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00088508100000000002 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005608+0000", 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.247e-06 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030447+0000", 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.024839515 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030545+0000", 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7453000000000004e-05 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.053649+0000", 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023104197999999999 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:295 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.692382428999998, 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.049664036, 2024-09-15T03:41:56.718 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 295 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004732+0000", 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004733+0000", 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0100000000000004e-07 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004737+0000", 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9049999999999999e-06 2024-09-15T03:41:56.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004745+0000", 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7540000000000006e-06 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004752+0000", 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9000000000000006e-06 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005631+0000", 2024-09-15T03:41:56.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087836100000000003 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005632+0000", 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.47e-07 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030868+0000", 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0252366669999999 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.030963+0000", 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4665000000000003e-05 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054396+0000", 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023433236 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.723 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:296 3.19s0 3:9a083222:::smithi10536496-8:head [read 0~983040,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:56.724 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:56.724 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 40.692356474, 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0501537240000001, 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 296 2024-09-15T03:41:56.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004758+0000", 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004759+0000", 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.82e-07 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004762+0000", 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2629999999999999e-06 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004769+0000", 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0260000000000002e-06 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.004780+0000", 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0569e-05 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005657+0000", 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087720500000000004 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:16.005658+0000", 2024-09-15T03:41:56.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.159e-06 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031236+0000", 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0255778339999999 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.031320+0000", 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4217999999999999e-05 2024-09-15T03:41:56.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:17.054912+0000", 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023591867999999998 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:412 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~688128] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.613646956, 2024-09-15T03:41:56.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013464116999999999, 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 412 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083468+0000", 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.083474+0000", 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2820000000000002e-06 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084200+0000", 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00072616300000000005 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084223+0000", 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2501e-05 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084230+0000", 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8709999999999997e-06 2024-09-15T03:41:56.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084286+0000", 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5943000000000003e-05 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.084297+0000", 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1009000000000001e-05 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.089736+0000", 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054388459999999998 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.089847+0000", 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000110355 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.739 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.090995+0000", 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011485 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094915+0000", 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039203249999999997 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.096868+0000", 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019529199999999999 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.096932+0000", 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3402e-05 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:413 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~540672] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:56.741 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.610473762000002, 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012499748, 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 413 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086641+0000", 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.086643+0000", 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0509999999999999e-06 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087150+0000", 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00050739100000000005 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087172+0000", 2024-09-15T03:41:56.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1903e-05 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087182+0000", 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4860000000000003e-06 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087222+0000", 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0246000000000001e-05 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087227+0000", 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0019999999999997e-06 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.091754+0000", 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045271299999999999 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.091849+0000", 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4918000000000002e-05 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.092538+0000", 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00068922000000000005 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095065+0000", 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025265919999999998 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099059+0000", 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039940599999999998 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099141+0000", 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1749000000000006e-05 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:414 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~229376] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.609204506000001, 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01143303, 2024-09-15T03:41:56.749 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 414 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087910+0000", 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.087911+0000", 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2190000000000001e-06 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088148+0000", 2024-09-15T03:41:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023696200000000001 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088161+0000", 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3091000000000001e-05 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.752 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088167+0000", 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.48e-06 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088196+0000", 2024-09-15T03:41:56.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9147999999999999e-05 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088200+0000", 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8060000000000001e-06 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093070+0000", 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048699090000000004 2024-09-15T03:41:56.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093181+0000", 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000111192 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.093854+0000", 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00067265900000000004 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095066+0000", 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001212762 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099287+0000", 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004220985 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.099343+0000", 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5816999999999998e-05 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:415 3.19s0 3:9992a557:::smithi10536496-15:head [setxattr _header (57) in=7b] snapc 18=[18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.608939639999999, 2024-09-15T03:41:56.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012852831, 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 415 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088175+0000", 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9399999999999995e-07 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088178+0000", 2024-09-15T03:41:56.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7439999999999999e-06 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088183+0000", 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1170000000000003e-06 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088186+0000", 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1109999999999999e-06 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088229+0000", 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2979999999999998e-05 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088237+0000", 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094281+0000", 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060443909999999997 2024-09-15T03:41:56.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094345+0000", 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3138000000000004e-05 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094761+0000", 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041665300000000001 2024-09-15T03:41:56.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095343+0000", 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058145699999999996 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.100956+0000", 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056129270000000002 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101028+0000", 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2167000000000002e-05 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.765 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:416 3.19s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 29.608924194, 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013816281999999999, 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 416 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088190+0000", 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088191+0000", 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9900000000000002e-07 2024-09-15T03:41:56.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088194+0000", 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5140000000000002e-06 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088201+0000", 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8070000000000002e-06 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088204+0000", 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2389999999999998e-06 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088260+0000", 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5937999999999999e-05 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.088266+0000", 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6330000000000001e-06 2024-09-15T03:41:56.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.094947+0000", 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066809970000000002 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.095006+0000", 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8712000000000003e-05 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101269+0000", 2024-09-15T03:41:56.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062632080000000001 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.101315+0000", 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6714e-05 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:27.102007+0000", 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000691221 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:423 3.19s0 3:9992a557:::smithi10536496-15:head [append 0~704512] snapc 19=[19,18,17,10] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.591852201999998, 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011973101, 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.772 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 423 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105262+0000", 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.105264+0000", 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.928e-06 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106293+0000", 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010284280000000001 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106312+0000", 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8940000000000002e-05 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106320+0000", 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3610000000000001e-06 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106383+0000", 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.106483+0000", 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9992000000000001e-05 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.107864+0000", 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013807120000000001 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.109771+0000", 2024-09-15T03:41:56.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019068659999999999 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.117186+0000", 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074148460000000001 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:28.117235+0000", 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9840999999999999e-05 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:448 3.as0 3:5dde9339:::smithi10536496-16:head [rollback 17] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.657464274999999, 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012127656000000001, 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 448 2024-09-15T03:41:56.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039650+0000", 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039651+0000", 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5000000000000003e-07 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039654+0000", 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1209999999999998e-06 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039661+0000", 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4830000000000002e-06 2024-09-15T03:41:56.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039664+0000", 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.016e-06 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041239+0000", 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015745449999999999 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041330+0000", 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1223000000000004e-05 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.041870+0000", 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00053994899999999996 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.043742+0000", 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018719940000000001 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.051733+0000", 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079906860000000003 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.051778+0000", 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5089000000000001e-05 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.783 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:449 3.as0 3:5dde9339:::smithi10536496-16:head [append 0~0] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.657446578999998, 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018300219999999999, 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 449 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039668+0000", 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.89e-07 2024-09-15T03:41:56.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039671+0000", 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2910000000000002e-06 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039676+0000", 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.383e-06 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.039680+0000", 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4879999999999999e-06 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042268+0000", 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025880899999999999 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042414+0000", 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000146312 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042875+0000", 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00046076800000000002 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044451+0000", 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001575833 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.057930+0000", 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013479326 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.057968+0000", 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.824e-05 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/51 ECSubWriteReply(tid=49, last_complete=65'44, committed=1, applied=1))", 2024-09-15T03:41:56.789 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.655437987999999, 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.47081936099999999, 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041677+0000", 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0900000000000001e-07 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.041681+0000", 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8970000000000001e-06 2024-09-15T03:41:56.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512389+0000", 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.470707388 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512401+0000", 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2897999999999999e-05 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512468+0000", 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6607999999999997e-05 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:32.512496+0000", 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7861e-05 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:625 3.as0 3:57d09725:::smithi10536496-48:head [append 0~475136] snapc 28=[28,27,26,25,24] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.668693360000001, 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011332543, 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 625 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.794 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028421+0000", 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028422+0000", 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2160000000000001e-06 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028856+0000", 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043326999999999998 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028866+0000", 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8009999999999998e-06 2024-09-15T03:41:56.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028871+0000", 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028939+0000", 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8388999999999999e-05 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029004+0000", 2024-09-15T03:41:56.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4908999999999998e-05 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.029559+0000", 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000555465 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030619+0000", 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010596799999999999 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039713+0000", 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090945100000000001 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.039754+0000", 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0303999999999999e-05 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:627 3.as0 3:57d09725:::smithi10536496-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-15T03:41:56.799 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.668209396, 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012208765, 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 627 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:56.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028905+0000", 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028906+0000", 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4700000000000002e-07 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028907+0000", 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.544e-06 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028911+0000", 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.028914+0000", 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:41:56.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030561+0000", 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016473379999999999 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.030614+0000", 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2911000000000002e-05 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.040187+0000", 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095729660000000005 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.040275+0000", 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7498999999999997e-05 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:43.041114+0000", 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000839151 2024-09-15T03:41:56.804 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/51 ECSubWriteReply(tid=88, last_complete=77'59, committed=1, applied=1))", 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.044436+0000", 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.652678183999999, 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.36005547199999999, 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044436+0000", 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044436+0000", 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044437+0000", 2024-09-15T03:41:56.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.04e-06 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.044442+0000", 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6909999999999996e-06 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404387+0000", 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.35994481 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404393+0000", 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8089999999999998e-06 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404446+0000", 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2889000000000001e-05 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.404492+0000", 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6233000000000003e-05 2024-09-15T03:41:56.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:719 3.1s0 3:80ab10c3:::smithi10536496-19:25 [read 0~245760,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:50.010483+0000", 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.6866316440000002, 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021205575000000001, 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.809 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 719 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010483+0000", 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010483+0000", 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010484+0000", 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2399999999999998e-07 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010488+0000", 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2069999999999997e-06 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010498+0000", 2024-09-15T03:41:56.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0132999999999999e-05 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010504+0000", 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.728e-06 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.011299+0000", 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00079533800000000003 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.011301+0000", 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1639999999999999e-06 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.026934+0000", 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015632384999999999 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.027099+0000", 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016540500000000001 2024-09-15T03:41:56.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.031689+0000", 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045895909999999996 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:720 3.1s0 3:80ab10c3:::smithi10536496-19:25 [read 0~245760,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:50.010509+0000", 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.6866050880000003, 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02157009, 2024-09-15T03:41:56.814 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 720 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010509+0000", 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010509+0000", 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010510+0000", 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010512+0000", 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.013e-06 2024-09-15T03:41:56.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010517+0000", 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0730000000000004e-06 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010521+0000", 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7309999999999999e-06 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.011331+0000", 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00081073099999999997 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.011332+0000", 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0020000000000001e-06 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.027474+0000", 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016141202 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.027578+0000", 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104456 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.032080+0000", 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045015419999999999 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:721 3.1s0 3:80ab10c3:::smithi10536496-19:25 [read 0~245760,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:50.010524+0000", 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 6.686590152, 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021878901999999999, 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 721 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:56.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010524+0000", 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010524+0000", 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010525+0000", 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.72e-07 2024-09-15T03:41:56.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010527+0000", 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2249999999999999e-06 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010534+0000", 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.464e-06 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.010538+0000", 2024-09-15T03:41:56.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9380000000000002e-06 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.011411+0000", 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00087231600000000004 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.011413+0000", 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7570000000000001e-06 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.027943+0000", 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016530955 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.028059+0000", 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011535399999999999 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:50.032403+0000", 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043445209999999996 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.824 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:56.825 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:56.825 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.826 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:41:56.827 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:56.825+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:56.927 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:56.927+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:56.929 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:56.929 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:56.929 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:41:56.929 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:56.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:41:57.029 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:41:57.028+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:41:57.030 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [3, 4] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 7, 11, 15, 3] 2024-09-15T03:41:57.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-15T03:41:57.044 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:57.044 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:41:57.045 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:41:57.045 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:41:57.045 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:41:57.046 INFO:tasks.rados.rados.0.smithi105.stdout:312: done (3 left) 2024-09-15T03:41:57.046 INFO:tasks.rados.rados.0.smithi105.stdout:314: done (2 left) 2024-09-15T03:41:57.046 INFO:tasks.rados.rados.0.smithi105.stdout:315: done (1 left) 2024-09-15T03:41:57.047 INFO:tasks.rados.rados.0.smithi105.stdout:316: done (0 left) 2024-09-15T03:41:57.047 INFO:tasks.rados.rados.0.smithi105.stdout:317: read oid 17 snap -1 2024-09-15T03:41:57.047 INFO:tasks.rados.rados.0.smithi105.stdout:317: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:57.047 INFO:tasks.rados.rados.0.smithi105.stdout:318: delete oid 19 current snap is 28 2024-09-15T03:41:57.056 INFO:tasks.rados.rados.0.smithi105.stdout:318: done (1 left) 2024-09-15T03:41:57.056 INFO:tasks.rados.rados.0.smithi105.stdout:319: snap_remove snap 23 2024-09-15T03:41:57.057 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:41:57.102 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:41:57.130 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:41:57.128+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.6s4 35/32 ECSubWrite(tid=0, reqid=client.4562.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.193532097000002, 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023454370000000001, 2024-09-15T03:41:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018415+0000", 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018416+0000", 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5300000000000004e-07 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018457+0000", 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1125999999999999e-05 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018674+0000", 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00021725800000000001 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018754+0000", 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0282000000000001e-05 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018806+0000", 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.018858+0000", 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1873000000000003e-05 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020727+0000", 2024-09-15T03:41:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001868897 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.020761+0000", 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4165999999999998e-05 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.141003349999998, 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084043045999999996, 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070945+0000", 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.124e-06 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070959+0000", 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4212999999999999e-05 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154508+0000", 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083548836000000001 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154515+0000", 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154566+0000", 2024-09-15T03:41:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1025e-05 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154578+0000", 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154965+0000", 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038690300000000002 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154987+0000", 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2172999999999999e-05 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=12, reqid=client.4562.0:150, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.057703091, 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035676890000000002, 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154245+0000", 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154487+0000", 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000242308 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156728+0000", 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022401589999999998 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156736+0000", 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3259999999999996e-06 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156781+0000", 2024-09-15T03:41:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4583000000000003e-05 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156789+0000", 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8359999999999998e-06 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157761+0000", 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097187399999999996 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157812+0000", 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0695999999999997e-05 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=14, reqid=client.4562.0:152, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.054759201000003, 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028494564, 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157189+0000", 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3399999999999998e-07 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157206+0000", 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7207e-05 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185008+0000", 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027801691999999999 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185015+0000", 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185068+0000", 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2333000000000002e-05 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185077+0000", 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0259999999999996e-06 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185660+0000", 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058343299999999996 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185683+0000", 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714e-05 2024-09-15T03:41:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=15, reqid=client.4562.0:164, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.027279432, 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027286849999999998, 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184668+0000", 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184669+0000", 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000005e-07 2024-09-15T03:41:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.184986+0000", 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000317027 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015847960000000001 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186578+0000", 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3220000000000004e-06 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186619+0000", 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1186000000000002e-05 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186629+0000", 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:41:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187377+0000", 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00074841899999999995 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.187397+0000", 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9233999999999999e-05 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=17, reqid=client.4562.0:166, at_version=35'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.02506245, 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099130431000000005, 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186886+0000", 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186906+0000", 2024-09-15T03:41:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0482000000000001e-05 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285393+0000", 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098487125999999994 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285401+0000", 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3780000000000003e-06 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285446+0000", 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028999999999999e-05 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285453+0000", 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285995+0000", 2024-09-15T03:41:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00054192900000000002 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.286015+0000", 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0579000000000001e-05 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 46.190434265, 2024-09-15T03:41:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030110250000000001, 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8299999999999998e-07 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022295+0000", 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078131100000000003 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022331+0000", 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6103000000000003e-05 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022338+0000", 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6410000000000002e-06 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022426+0000", 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7366000000000003e-05 2024-09-15T03:41:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022437+0000", 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.131e-05 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024502+0000", 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002065271 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024524+0000", 2024-09-15T03:41:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1739999999999999e-05 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 47/32 ECSubWrite(tid=66, reqid=osd.11.0:66, at_version=47'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 43.203416341999997, 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003545678, 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:41:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.29e-07 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008562+0000", 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0871999999999999e-05 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008606+0000", 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4057999999999997e-05 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008618+0000", 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878999999999999e-05 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008722+0000", 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104236 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008732+0000", 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0142999999999999e-05 2024-09-15T03:41:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012049+0000", 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033168920000000001 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012077+0000", 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7368999999999998e-05 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 39.187407209, 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042098190000000001, 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9600000000000001e-07 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024564+0000", 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3714e-05 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024603+0000", 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9048000000000003e-05 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024610+0000", 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1520000000000003e-06 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024717+0000", 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000106872 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024728+0000", 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028684+0000", 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039562190000000004 2024-09-15T03:41:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028750+0000", 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5658000000000001e-05 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.18s6 51/49 ECSubWrite(tid=153, reqid=client.4562.0:369, at_version=51'11, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 39.176717011999997, 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028221240000000001, 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035230+0000", 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.035231+0000", 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4699999999999998e-07 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036344+0000", 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001113721 2024-09-15T03:41:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036395+0000", 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0247000000000002e-05 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036403+0000", 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4630000000000001e-06 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036467+0000", 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3876999999999998e-05 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.036480+0000", 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2818e-05 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.038029+0000", 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015492629999999999 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.038052+0000", 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3487999999999999e-05 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 56/32 ECSubWrite(tid=88, reqid=osd.11.0:88, at_version=56'26, trim_to=0'0, roll_forward_to=51'24))", 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 34.148728961000003, 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003747588, 2024-09-15T03:41:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063219+0000", 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063262+0000", 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3173000000000002e-05 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063457+0000", 2024-09-15T03:41:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019513700000000001 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063464+0000", 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6120000000000004e-06 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063517+0000", 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3477999999999999e-05 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063528+0000", 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.112e-05 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066940+0000", 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034120299999999999 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066966+0000", 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5724e-05 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 62/32 ECSubWrite(tid=60, reqid=osd.2.0:60, at_version=62'32, trim_to=0'0, roll_forward_to=58'30))", 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.169756760999999, 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025058089999999999, 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042191+0000", 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042217+0000", 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5256e-05 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042269+0000", 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2383999999999999e-05 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042281+0000", 2024-09-15T03:41:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1977000000000001e-05 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042333+0000", 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2537999999999997e-05 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.042345+0000", 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1307e-05 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044670+0000", 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023257120000000002 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:29.044696+0000", 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5996e-05 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 23.156275925999999, 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025217009999999999, 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055671+0000", 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.055672+0000", 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.18e-07 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056050+0000", 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00037845499999999998 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056164+0000", 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113744 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056168+0000", 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9770000000000002e-06 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056220+0000", 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2176999999999999e-05 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.056230+0000", 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8500000000000006e-06 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058166+0000", 2024-09-15T03:41:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001936243 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:34.058193+0000", 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7137000000000001e-05 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 77/71 ECSubWrite(tid=149, reqid=osd.4.0:149, at_version=77'31, trim_to=0'0, roll_forward_to=75'29))", 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:41:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.189605916, 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021045639000000001, 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:41:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022342+0000", 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022376+0000", 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4474999999999998e-05 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042674+0000", 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020297453 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042677+0000", 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5939999999999998e-06 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042725+0000", 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7528000000000003e-05 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042736+0000", 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1420000000000001e-05 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.043360+0000", 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00062334599999999999 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.043387+0000", 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7421000000000001e-05 2024-09-15T03:41:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(osd.5.22:4 3.5s0 3:a7103f89:::smithi10536496-39:head [assert-version v41,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.170447304, 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097123769999999995, 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041501+0000", 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.218e-06 2024-09-15T03:41:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041508+0000", 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.421e-06 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041525+0000", 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7164e-05 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041531+0000", 2024-09-15T03:41:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9279999999999998e-06 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041589+0000", 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8352e-05 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041750+0000", 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000160657 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.051212+0000", 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094626369999999994 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpReadReply(3.5s0 77/72 ECSubReadReply(tid=4, attrs_read=0))", 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.164743881, 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034924600000000002, 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047204+0000", 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4000000000000001e-07 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050148+0000", 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029437109999999999 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050597+0000", 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044960600000000002 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050605+0000", 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.695e-06 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050651+0000", 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5874000000000002e-05 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050696+0000", 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4934e-05 2024-09-15T03:41:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 78/71 ECSubWrite(tid=91, reqid=osd.2.0:91, at_version=78'61, trim_to=0'0, roll_forward_to=77'59))", 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.18454011, 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002303907, 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027407+0000", 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0800000000000001e-07 2024-09-15T03:41:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027432+0000", 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4953e-05 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027482+0000", 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9756999999999998e-05 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027493+0000", 2024-09-15T03:41:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1285e-05 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027561+0000", 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7124000000000003e-05 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.027571+0000", 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0756e-05 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.029686+0000", 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021143490000000002 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.029711+0000", 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5474999999999999e-05 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 78/71 ECSubWrite(tid=67, reqid=osd.14.0:67, at_version=78'46, trim_to=0'0, roll_forward_to=77'44))", 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.183626586999999, 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002840736, 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028321+0000", 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5900000000000001e-07 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028364+0000", 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3287000000000001e-05 2024-09-15T03:41:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028453+0000", 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8350999999999995e-05 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028460+0000", 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4959999999999999e-06 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028524+0000", 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3696999999999997e-05 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.028536+0000", 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2666999999999999e-05 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031128+0000", 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025911660000000002 2024-09-15T03:41:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031162+0000", 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3912999999999999e-05 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 78/71 ECSubWrite(tid=92, reqid=osd.2.0:92, at_version=78'63, trim_to=0'0, roll_forward_to=77'59))", 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:41:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.180710172, 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003315836, 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031238+0000", 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0669999999999999e-06 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031262+0000", 2024-09-15T03:41:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4187999999999999e-05 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031302+0000", 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9521999999999999e-05 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031309+0000", 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0489999999999998e-06 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032119+0000", 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080987699999999997 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032128+0000", 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2089999999999994e-06 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.034523+0000", 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023950349999999998 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.034553+0000", 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9889000000000001e-05 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 78/71 ECSubWrite(tid=151, reqid=osd.4.0:151, at_version=78'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 12.179732111, 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043349979999999996, 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.35e-07 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032236+0000", 2024-09-15T03:41:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0922000000000001e-05 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032321+0000", 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032327+0000", 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.516e-06 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032717+0000", 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038967199999999998 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032727+0000", 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.876e-06 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036523+0000", 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037960239999999998 2024-09-15T03:41:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036550+0000", 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7307000000000001e-05 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:41:57.308 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:41:57.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:41:57.310 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:57.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:57.311 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:57.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:57.311 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:57.312 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:57.312 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:57.312 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:57.313 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:57.309+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:57.319 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:57.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:57.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:57.361 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:57.361 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:57.381 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:57.401 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:57.401 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:57.410 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:57.409+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:57.421 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:57.427 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:57.427 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:57.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:41:57.441 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:57.462 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:57.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:41:57.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:41:57.502 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:41:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:41:57.511 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:57.511+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:57.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:57.518+0000 7fdeb1d9b640 0 -- 172.21.15.5:0/2811472613 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7fdeb41952e0 msgr2=0x7fdeb41976c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-15T03:41:57.527 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:41:57.527 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:41:57.548 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:41:57.548 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:41:57.581 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:57.582 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:57.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:41:57.612 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:41:57.611+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:41:57.692 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:57.692 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:57.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:41:57.713 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:57.712+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:57.802 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:57.803 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:57.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:41:57.814 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:57.813+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:57.912 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:57.912 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:57.914 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:57.914+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:57.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:41:58.016 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:58.014+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:41:58.049 INFO:tasks.rados.rados.0.smithi105.stdout:319: done (1 left) 2024-09-15T03:41:58.049 INFO:tasks.rados.rados.0.smithi105.stdout:320: snap_create 2024-09-15T03:41:58.049 INFO:tasks.rados.rados.0.smithi105.stdout:317: expect (ObjNum 102 snap 16 seq_num 102) 2024-09-15T03:41:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-09-15T03:41:58.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-15T03:41:58.116 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:58.116+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:58.141 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.141 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:41:58.217 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:41:58.216+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:41:58.250 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.250 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:41:58.318 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:58.317+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:58.361 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.361 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:41:58.419 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:41:58.417+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:41:58.512 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.512 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.520 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:41:58.519+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:41:58.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:41:58.621 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:58.620+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:58.663 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.664 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:41:58.722 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:41:58.720+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:41:58.775 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.775 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.783 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:41:58.829 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:58.828+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:58.899 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:58.899 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:58.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:41:58.950 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:58.949+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:59.008 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-09-15T03:41:59.009 INFO:tasks.rados.rados.0.smithi105.stdout:317: done (1 left) 2024-09-15T03:41:59.009 INFO:tasks.rados.rados.0.smithi105.stdout:320: done (0 left) 2024-09-15T03:41:59.010 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 13 current snap is 29 2024-09-15T03:41:59.010 INFO:tasks.rados.rados.0.smithi105.stdout:321: seq_num 127 ranges {0=589824,589824=753664,1343488=475136,1818624=327680} 2024-09-15T03:41:59.012 INFO:tasks.rados.rados.0.smithi105.stdout:321: writing smithi10536496-13 from 0 to 589824 tid 1 2024-09-15T03:41:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:321: writing smithi10536496-13 from 589824 to 1343488 tid 2 2024-09-15T03:41:59.018 INFO:tasks.rados.rados.0.smithi105.stdout:321: writing smithi10536496-13 from 1343488 to 1818624 tid 3 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:321: writing smithi10536496-13 from 1818624 to 2146304 tid 4 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:322: copy_from oid 20 from oid 31 current snap is 29 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing write tid 1 to smithi10536496-13 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:323: read oid 1 snap -1 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:323: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:324: read oid 19 snap -1 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:324: expect deleted 2024-09-15T03:41:59.021 INFO:tasks.rados.rados.0.smithi105.stdout:325: snap_remove snap 28 2024-09-15T03:41:59.026 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:41:59.025+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:41:59.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-15T03:41:59.059 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.059 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.064 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:41:59.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:41:59.126 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:41:59.126+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:41:59.166 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.166 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:41:59.227 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:59.226+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:59.273 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.273 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:41:59.328 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:59.327+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:59.421 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.421 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.429 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:41:59.428+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:41:59.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:41:59.526 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.526 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.530 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:41:59.529+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:41:59.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:41:59.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:41:59.630 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:41:59.629+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:41:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:41:59.687 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:41:59.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:41:59.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:41:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:41:59.729 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:41:59.729 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:41:59.731 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:41:59.730+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:41:59.749 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:41:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:41:59.768 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:41:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:41:59.789 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:41:59.789 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:41:59.809 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:41:59.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:41:59.818 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:41:59.818 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:41:59.827 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:41:59.830 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:41:59.830 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:41:59.833 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:41:59.832+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:41:59.849 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:41:59.850 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:41:59.870 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:41:59.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:41:59.889 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:41:59.889 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:41:59.909 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:41:59.910 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.9s6 34/32 ECSubWrite(tid=1, reqid=client.4562.0:22, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 57.905003031, 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008773889, 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019893+0000", 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8900000000000001e-07 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019951+0000", 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7080000000000002e-05 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020259+0000", 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030882000000000001 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020269+0000", 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4690000000000003e-06 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020353+0000", 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4121000000000003e-05 2024-09-15T03:41:59.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020367+0000", 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.401e-05 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028603+0000", 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082364340000000008 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028667+0000", 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3665999999999997e-05 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=64, reqid=client.4562.0:284, at_version=0'0, trim_to=0'0, roll_forward_to=45'28))", 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.886280939999999, 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090921019999999995, 2024-09-15T03:41:59.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038615+0000", 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038616+0000", 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0240000000000001e-06 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038629+0000", 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2891000000000001e-05 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038669+0000", 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9725000000000003e-05 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038682+0000", 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3095000000000001e-05 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038746+0000", 2024-09-15T03:41:59.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3850999999999996e-05 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.038758+0000", 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2329e-05 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047686+0000", 2024-09-15T03:41:59.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089280980000000006 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.047707+0000", 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1089000000000001e-05 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:300 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~442368] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.915653104, 2024-09-15T03:41:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0389183930000001, 2024-09-15T03:41:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:41:59.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009245+0000", 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.556e-06 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009830+0000", 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058575800000000003 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009844+0000", 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3247e-05 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009851+0000", 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4850000000000003e-06 2024-09-15T03:41:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010438+0000", 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058667099999999998 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010441+0000", 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2119999999999999e-06 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031346+0000", 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0209051899999999 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031477+0000", 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130712 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.032397+0000", 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091970699999999997 2024-09-15T03:41:59.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037882+0000", 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054851639999999998 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048077+0000", 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010194896 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048161+0000", 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4795000000000004e-05 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:301 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~720896] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.915028520999996, 2024-09-15T03:41:59.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456429519999999, 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 301 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:59.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010594+0000", 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072566600000000003 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010605+0000", 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1228e-05 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010611+0000", 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8390000000000003e-06 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010679+0000", 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7966999999999995e-05 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010681+0000", 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1409999999999999e-06 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033259+0000", 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.022577372 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033385+0000", 2024-09-15T03:41:59.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012635200000000001 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.034673+0000", 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001287853 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038109+0000", 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003436508 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055462+0000", 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017353200999999999 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055511+0000", 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8217999999999998e-05 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:302 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~49152] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:59.954 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.91427814, 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.045100688, 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 302 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.955 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010619+0000", 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0699999999999996e-07 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010663+0000", 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552000000000003e-05 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010680+0000", 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6175999999999999e-05 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010685+0000", 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8619999999999999e-06 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010726+0000", 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0259000000000002e-05 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010727+0000", 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.753e-06 2024-09-15T03:41:59.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035776+0000", 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0250488879999999 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035909+0000", 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013254900000000001 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.036628+0000", 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00071956899999999998 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038283+0000", 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016543490000000001 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055650+0000", 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017367301000000002 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055719+0000", 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8622999999999997e-05 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:303 3.11s0 3:8f673fa9:::smithi10536496-24:head [setxattr _header (56) in=7b] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.914200254000001, 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0452317200000001, 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 303 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010697+0000", 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010700+0000", 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2509999999999999e-06 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010706+0000", 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4989999999999999e-06 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010711+0000", 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2250000000000002e-06 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010765+0000", 2024-09-15T03:41:59.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4784000000000003e-05 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010767+0000", 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4160000000000001e-06 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037038+0000", 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.026271521 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037181+0000", 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014324800000000001 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037675+0000", 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049306399999999998 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.039474+0000", 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017996729999999999 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055837+0000", 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016362393999999999 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055928+0000", 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1051000000000006e-05 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:304 3.11s0 3:8f673fa9:::smithi10536496-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.914181331999998, 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.050398215, 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 304 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-09-15T03:41:59.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010717+0000", 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.125e-06 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010724+0000", 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6880000000000002e-06 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010728+0000", 2024-09-15T03:41:59.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.777e-06 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010795+0000", 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7277e-05 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010796+0000", 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037918+0000", 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.02712114 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038025+0000", 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010774 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056315+0000", 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018289716000000001 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056417+0000", 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102336 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061113+0000", 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046956749999999998 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:317 3.11s0 3:8b799747:::smithi10536496-30:head [rollback e] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.865815857999998, 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010406076, 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 317 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059081+0000", 2024-09-15T03:41:59.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.151e-06 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059096+0000", 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4499999999999997e-06 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060096+0000", 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010001019999999999 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060189+0000", 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2320999999999999e-05 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060639+0000", 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044999599999999998 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062024+0000", 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013859650000000001 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069431+0000", 2024-09-15T03:41:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074061190000000001 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069486+0000", 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5831999999999997e-05 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:318 3.11s0 3:8b799747:::smithi10536496-30:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:59.979 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.865794821999998, 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010504207999999999, 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 318 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059102+0000", 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6600000000000002e-07 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059104+0000", 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.215e-06 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059110+0000", 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0279999999999997e-06 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059114+0000", 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3340000000000003e-06 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061223+0000", 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002108249 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061279+0000", 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6418000000000001e-05 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061697+0000", 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000418109 2024-09-15T03:41:59.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062208+0000", 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051137000000000003 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069565+0000", 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073569799999999999 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069606+0000", 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0139000000000003e-05 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:374 3.11s0 3:8b799747:::smithi10536496-30:head [delete] snapc e=[c,b] ondisk+write+known_if_redirected+full_force+supports_pool_eio e51)", 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.894488453999998, 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086570359999999999, 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 374 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030408+0000", 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030410+0000", 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1500000000000002e-06 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030418+0000", 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5989999999999996e-06 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030442+0000", 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4612000000000001e-05 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030450+0000", 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2959999999999999e-06 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030513+0000", 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.030644+0000", 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130591 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.031027+0000", 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038316199999999999 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.031678+0000", 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00065064699999999997 2024-09-15T03:41:59.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.039008+0000", 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073303159999999999 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:18.039065+0000", 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6904000000000003e-05 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds3 71/51 ECSubWrite(tid=49, reqid=client.4562.0:534, at_version=71'27, trim_to=0'0, roll_forward_to=71'24))", 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.052061+0000", 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.872835378000001, 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084783219999999999, 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052061+0000", 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052061+0000", 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052062+0000", 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.99e-07 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052186+0000", 2024-09-15T03:41:59.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012453799999999999 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052221+0000", 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4724999999999997e-05 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052228+0000", 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1860000000000003e-06 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052275+0000", 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6860999999999997e-05 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052286+0000", 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1223000000000001e-05 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.060511+0000", 2024-09-15T03:41:59.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082247929999999993 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.060539+0000", 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8297e-05 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:542 3.11s0 3:8f673fa9:::smithi10536496-24:head [setxattr _header (57) in=7b] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.812584396999998, 2024-09-15T03:41:59.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83759245800000004, 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 542 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7700000000000004e-07 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:41:59.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112315+0000", 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7999999999999999e-06 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112323+0000", 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5630000000000003e-06 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112326+0000", 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.114769+0000", 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024421389999999999 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.114833+0000", 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4281000000000003e-05 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.115220+0000", 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038725499999999998 2024-09-15T03:41:59.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.116192+0000", 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097158599999999998 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.949795+0000", 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83360352100000001 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.949904+0000", 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109096 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:543 3.11s0 3:8f673fa9:::smithi10536496-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112335+0000", 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.812561031000001, 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83899146099999999, 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 543 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112335+0000", 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112335+0000", 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112336+0000", 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9900000000000001e-07 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112337+0000", 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.728e-06 2024-09-15T03:42:00.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112344+0000", 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2609999999999999e-06 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112347+0000", 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7390000000000001e-06 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.116073+0000", 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037256780000000001 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.116139+0000", 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6052999999999999e-05 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.950344+0000", 2024-09-15T03:42:00.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83420500900000005 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.950463+0000", 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118498 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.951327+0000", 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086409599999999996 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 77/32 ECSubWrite(tid=145, reqid=osd.11.0:145, at_version=77'40, trim_to=0'0, roll_forward_to=75'36))", 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.032545+0000", 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.892351357000001, 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010871377, 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032545+0000", 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032545+0000", 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032546+0000", 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032574+0000", 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8705999999999999e-05 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032606+0000", 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2181000000000002e-05 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032614+0000", 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9740000000000002e-06 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032942+0000", 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032719200000000002 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032952+0000", 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0693000000000001e-05 2024-09-15T03:42:00.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.043373+0000", 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010420469 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.043416+0000", 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3414000000000001e-05 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.es1 77/71 ECSubWrite(tid=89, reqid=osd.2.0:89, at_version=77'59, trim_to=0'0, roll_forward_to=75'55))", 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.033325+0000", 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.891571300000001, 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010145891000000001, 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033325+0000", 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033325+0000", 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033326+0000", 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4799999999999997e-07 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033347+0000", 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0803999999999999e-05 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033392+0000", 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5757000000000003e-05 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033403+0000", 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0825999999999999e-05 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033465+0000", 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1544999999999994e-05 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033476+0000", 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.135e-05 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.043450+0000", 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099743279999999993 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.043471+0000", 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0432999999999999e-05 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.es1 78/71 ECSubWrite(tid=92, reqid=osd.2.0:92, at_version=78'63, trim_to=0'0, roll_forward_to=77'59))", 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:45.029299+0000", 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.895597014, 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093767680000000006, 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029299+0000", 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029299+0000", 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029300+0000", 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4799999999999997e-07 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029324+0000", 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4286000000000001e-05 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029373+0000", 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8671999999999997e-05 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029380+0000", 2024-09-15T03:42:00.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1620000000000002e-06 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029429+0000", 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8488000000000001e-05 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.029439+0000", 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0547000000000001e-05 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.038644+0000", 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092044350000000004 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.038676+0000", 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2329999999999997e-05 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.023 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 78/32 ECSubWrite(tid=148, reqid=osd.11.0:148, at_version=78'42, trim_to=0'0, roll_forward_to=77'40))", 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:45.030046+0000", 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.894850045, 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0086890030000000007, 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030046+0000", 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030046+0000", 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030046+0000", 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0800000000000001e-07 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030069+0000", 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3022000000000001e-05 2024-09-15T03:42:00.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030218+0000", 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000148942 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030224+0000", 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.156e-06 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030294+0000", 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9432999999999997e-05 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030304+0000", 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.039e-05 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.038692+0000", 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008387205 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.038735+0000", 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3646999999999997e-05 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.13s6 78/72 ECSubWrite(tid=147, reqid=client.4562.0:642, at_version=78'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:45.030072+0000", 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.894824581, 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099077220000000008, 2024-09-15T03:42:00.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030072+0000", 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030072+0000", 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030072+0000", 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030136+0000", 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3904000000000003e-05 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030264+0000", 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012814099999999999 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030270+0000", 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030321+0000", 2024-09-15T03:42:00.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1396000000000001e-05 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.030333+0000", 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1732999999999999e-05 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.039953+0000", 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096197490000000004 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.039979+0000", 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6777000000000001e-05 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 78/32 ECSubWrite(tid=149, reqid=osd.11.0:149, at_version=78'44, trim_to=0'0, roll_forward_to=77'40))", 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:45.031193+0000", 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.893703289999999, 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090041540000000003, 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.031193+0000", 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.031193+0000", 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.031194+0000", 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.64e-07 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.031247+0000", 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2688000000000001e-05 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.031301+0000", 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4265000000000003e-05 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.031308+0000", 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.951e-06 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.033455+0000", 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021470880000000001 2024-09-15T03:42:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.033465+0000", 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0282000000000001e-05 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.040171+0000", 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067056620000000003 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.040197+0000", 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6254e-05 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 84/32 ECSubWrite(tid=174, reqid=client.4562.0:723, at_version=84'55, trim_to=0'0, roll_forward_to=81'52))", 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8983394429999993, 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010574913, 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.038 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-07 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027143+0000", 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058585600000000005 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027185+0000", 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2546000000000002e-05 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027194+0000", 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3690000000000007e-06 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027258+0000", 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3995e-05 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027269+0000", 2024-09-15T03:42:00.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1503e-05 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.037103+0000", 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098336789999999997 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.037132+0000", 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8844999999999999e-05 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.043 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:00.044 INFO:tasks.rados.rados.0.smithi105.stdout:325: done (4 left) 2024-09-15T03:42:00.044 INFO:tasks.rados.rados.0.smithi105.stdout:326: snap_remove snap 18 2024-09-15T03:42:00.044 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:00.045 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:00.043+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:00.045 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:00.045 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:00.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:42:00.046 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:42:00.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:41:59.954+0000 7f8402ffd640 0 -- 172.21.15.5:0/1335373556 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f84041951f0 msgr2=0x7f84041975d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:00.047 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:00.047 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:42:00.047 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:00.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:00.048 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:42:00.049 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-09-15T03:42:00.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-15T03:42:00.145 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:00.144+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:00.149 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:42:00.149 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:00.158 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:42:00.246 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:00.245+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:00.257 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:42:00.257 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:42:00.257 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:00.266 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:42:00.348 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:00.347+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:00.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.382985658000003, 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:42:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:42:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:42:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.376378410999997, 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:42:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.376360796999997, 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:42:00.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.371635591999997, 2024-09-15T03:42:00.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:42:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.431 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.432 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:42:00.432 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.359053848000002, 2024-09-15T03:42:00.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:42:00.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:42:00.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:42:00.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:42:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.359042305999999, 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.339850679000001, 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:42:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:42:00.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.339366783999999, 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.451 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:42:00.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.339325141000003, 2024-09-15T03:42:00.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:42:00.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.337071694000002, 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:42:00.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:42:00.465 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.336083352999999, 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:42:00.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.279703003000002, 2024-09-15T03:42:00.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036705979, 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125266+0000", 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125267+0000", 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.68e-07 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125270+0000", 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4709999999999999e-06 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161517+0000", 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.036246426999999998 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161523+0000", 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161562+0000", 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8571000000000001e-05 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161972+0000", 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041002299999999997 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.243600903999997, 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:42:00.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:42:00.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:42:00.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 58.239633240000003, 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:00.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:287 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 8] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 48.376689651, 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.820123202, 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 287 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.483 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028281+0000", 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5200000000000001e-06 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028286+0000", 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028300+0000", 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3668e-05 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028307+0000", 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9539999999999998e-06 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029951+0000", 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001644419 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029955+0000", 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6260000000000001e-06 2024-09-15T03:42:00.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037676+0000", 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077205920000000001 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037910+0000", 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234212 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057184+0000", 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019273795999999999 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.059033+0000", 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00184902 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848316+0000", 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78928352999999996 2024-09-15T03:42:00.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848403+0000", 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6583999999999996e-05 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 45/44 ECSubWriteReply(tid=39, last_complete=45'16, committed=1, applied=1))", 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 48.340018292000003, 2024-09-15T03:42:00.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78386887299999997, 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:00.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064956+0000", 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848087+0000", 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78313173400000002 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848099+0000", 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1173e-05 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848179+0000", 2024-09-15T03:42:00.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0921999999999995e-05 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848820+0000", 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064024899999999996 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds2 71/51 ECSubWrite(tid=50, reqid=client.4562.0:535, at_version=71'28, trim_to=0'0, roll_forward_to=71'24))", 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.052671+0000", 2024-09-15T03:42:00.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.352297872000001, 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.230093292, 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052671+0000", 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052671+0000", 2024-09-15T03:42:00.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052672+0000", 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052692+0000", 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9967e-05 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282058+0000", 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.22936631900000001 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282064+0000", 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6330000000000001e-06 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282143+0000", 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9660999999999995e-05 2024-09-15T03:42:00.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282152+0000", 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8580000000000006e-06 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282739+0000", 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058726799999999995 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282765+0000", 2024-09-15T03:42:00.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5075e-05 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 83/82 ECSubWriteReply(tid=81, last_complete=83'36, committed=1, applied=1))", 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:50.039554+0000", 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.365415171, 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.41583952099999999, 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039554+0000", 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039554+0000", 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039555+0000", 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-09-15T03:42:00.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039558+0000", 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.454878+0000", 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.41532027900000001 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.454883+0000", 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7149999999999997e-06 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.454949+0000", 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6211999999999996e-05 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.455393+0000", 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000444289 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 83/82 ECSubWriteReply(tid=83, last_complete=83'36, committed=1, applied=1))", 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:50.042724+0000", 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.362245033000001, 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.96124359599999998, 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042724+0000", 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042724+0000", 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042725+0000", 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2500000000000005e-07 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042730+0000", 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0549999999999999e-06 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003879+0000", 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.96114944499999999 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003884+0000", 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8400000000000002e-06 2024-09-15T03:42:00.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003941+0000", 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6838e-05 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003968+0000", 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6693e-05 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s1 83/82 ECSubWrite(tid=11, reqid=client.4562.0:713, at_version=83'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.17400638, 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.26109879699999999, 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.231156+0000", 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019295299999999999 2024-09-15T03:42:00.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490877+0000", 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.25972105600000001 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490882+0000", 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5750000000000002e-06 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490951+0000", 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.923e-05 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490958+0000", 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8549999999999996e-06 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.492040+0000", 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010815460000000001 2024-09-15T03:42:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.492062+0000", 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1837000000000001e-05 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:00.510 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:00.513 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:42:00.514 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:00.512+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:00.617 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:00.612+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:00.623 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:42:00.624 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:00.633 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:42:00.714 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:00.713+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:00.731 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:00.731 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:00.732 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:42:00.732 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:42:00.732 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:00.740 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:42:00.816 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:00.815+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.831100356999997, 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.880 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:42:00.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.819471458000002, 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.884 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:42:00.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.889 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.801827957999997, 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:42:00.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:42:00.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:42:00.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.792002209000003, 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.895 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:42:00.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:42:00.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.899 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.762925176000003, 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:42:00.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:42:00.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.752008693000001, 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017309390000000001, 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:42:00.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126067+0000", 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126072+0000", 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3920000000000002e-06 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143252+0000", 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017179104000000001 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143257+0000", 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143315+0000", 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7958000000000003e-05 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143376+0000", 2024-09-15T03:42:00.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0535000000000002e-05 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=19, reqid=client.4562.0:144, at_version=35'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.731623540000001, 2024-09-15T03:42:00.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015149222, 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.85e-07 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146470+0000", 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7309999999999999e-05 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160975+0000", 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014505123 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160981+0000", 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161035+0000", 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3517999999999998e-05 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.065e-05 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161571+0000", 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052504699999999995 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161601+0000", 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:42:00.915 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=23, reqid=client.4562.0:154, at_version=35'12, trim_to=0'0, roll_forward_to=35'11))", 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.712696143000002, 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.083287300999999994, 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.916 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165380+0000", 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0999999999999997e-07 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165457+0000", 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6823999999999995e-05 2024-09-15T03:42:00.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247921+0000", 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082464198000000002 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247926+0000", 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9880000000000004e-06 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247954+0000", 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8306e-05 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247960+0000", 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7570000000000001e-06 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248650+0000", 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000689816 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248666+0000", 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6501999999999999e-05 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.921 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s1 3.1f172733 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.867211052000002, 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017692614, 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.922 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010866+0000", 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010929+0000", 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010944+0000", 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4701e-05 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010952+0000", 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4610000000000002e-06 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011020+0000", 2024-09-15T03:42:00.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8553000000000006e-05 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011023+0000", 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8150000000000002e-06 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028536+0000", 2024-09-15T03:42:00.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017512913 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028557+0000", 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0716e-05 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.927 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:310 3.as1 3:5629793d:::smithi10536496-26:head [delete] snapc e=[e,c,b] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.872615932999999, 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0408854540000001, 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 310 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.928 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005461+0000", 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.643e-06 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005465+0000", 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005480+0000", 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5407999999999999e-05 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005487+0000", 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4229999999999999e-06 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008383+0000", 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002895309 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008386+0000", 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8949999999999998e-06 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.031743+0000", 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0233571130000001 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.032058+0000", 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000315123 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.033215+0000", 2024-09-15T03:42:00.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011571579999999999 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.037843+0000", 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046277380000000002 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.045957+0000", 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081145560000000002 2024-09-15T03:42:00.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046345+0000", 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000387229 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 50/49 ECSubWrite(tid=151, reqid=client.4562.0:335, at_version=50'20, trim_to=0'0, roll_forward_to=50'18))", 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:42:00.935 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.747303873999996, 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067600538000000002, 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130773+0000", 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130954+0000", 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018089000000000001 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196934+0000", 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065979530999999994 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196943+0000", 2024-09-15T03:42:00.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197012+0000", 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8745999999999995e-05 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197025+0000", 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2909e-05 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198352+0000", 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001327188 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198372+0000", 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9933999999999999e-05 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.940 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:348 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.729784473999999, 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010390460000000001, 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.941 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 348 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148291+0000", 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.84e-07 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148293+0000", 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.858e-06 2024-09-15T03:42:00.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148299+0000", 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9780000000000002e-06 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.148303+0000", 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8249999999999998e-06 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.154925+0000", 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066227459999999997 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.155002+0000", 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6196e-05 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157677+0000", 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0026754650000000001 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.157794+0000", 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117167 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.158681+0000", 2024-09-15T03:42:00.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088684099999999997 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:532 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~442368] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.046777+0000", 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.831297957, 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011744466, 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.947 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 532 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.046777+0000", 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.046777+0000", 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.046779+0000", 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.562e-06 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047019+0000", 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024031700000000001 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047032+0000", 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3385e-05 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047038+0000", 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3149999999999999e-06 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047080+0000", 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2410000000000002e-05 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047153+0000", 2024-09-15T03:42:00.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2538000000000002e-05 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047668+0000", 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00051552100000000001 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.048400+0000", 2024-09-15T03:42:00.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073139699999999995 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.058451+0000", 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010051164 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.058522+0000", 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0857000000000003e-05 2024-09-15T03:42:00.953 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:533 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~524288] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.828039484000001, 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012117856, 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.954 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 533 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050163+0000", 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012692000000000001 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050173+0000", 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8099999999999992e-06 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050177+0000", 2024-09-15T03:42:00.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.676e-06 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050212+0000", 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4751e-05 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050272+0000", 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9998999999999999e-05 2024-09-15T03:42:00.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050798+0000", 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052661299999999999 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052872+0000", 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020738010000000001 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.062112+0000", 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092395299999999993 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.062153+0000", 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1879999999999999e-05 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:534 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~196608] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.051155+0000", 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.826919595, 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013798464999999999, 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.961 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 534 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051155+0000", 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051155+0000", 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051157+0000", 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1370000000000001e-06 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051316+0000", 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000159555 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051339+0000", 2024-09-15T03:42:00.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2555e-05 2024-09-15T03:42:00.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051345+0000", 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.229e-06 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051453+0000", 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010767700000000001 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051515+0000", 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2150999999999995e-05 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051939+0000", 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042459399999999998 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.054404+0000", 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024641580000000001 2024-09-15T03:42:00.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.064895+0000", 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010491097 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.064954+0000", 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9311999999999997e-05 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:535 3.ds0 3:b9f64cc8:::smithi10536496-42:head [setxattr _header (57) in=7b] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.826721521, 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.23179859899999999, 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 535 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:00.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051358+0000", 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3900000000000002e-06 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051364+0000", 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4710000000000004e-06 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051368+0000", 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7689999999999998e-06 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052278+0000", 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00091056999999999998 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052376+0000", 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7162000000000002e-05 2024-09-15T03:42:00.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052640+0000", 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026434599999999999 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.057117+0000", 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044771780000000001 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283090+0000", 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.22597297899999999 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283152+0000", 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2143000000000005e-05 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.973 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:536 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.051372+0000", 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.826702860000001, 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.232996072, 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 536 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051372+0000", 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051372+0000", 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051373+0000", 2024-09-15T03:42:00.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9999999999999998e-07 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051390+0000", 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7105999999999999e-05 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051399+0000", 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4709999999999996e-06 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051403+0000", 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.155e-06 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052799+0000", 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013954620000000001 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052867+0000", 2024-09-15T03:42:00.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8361000000000005e-05 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283472+0000", 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.23060489100000001 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283578+0000", 2024-09-15T03:42:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010643 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.284368+0000", 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00078979600000000005 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:634 3.ds0 3:b9f64cc8:::smithi10536496-42:28 [read 0~3555328,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:44.036221+0000", 2024-09-15T03:42:00.979 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.84185385, 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011783227, 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 634 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036221+0000", 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036221+0000", 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036222+0000", 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6199999999999997e-07 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.51e-06 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036234+0000", 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5739999999999998e-06 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036239+0000", 2024-09-15T03:42:00.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4599999999999996e-06 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036839+0000", 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00060048700000000003 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036922+0000", 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2769000000000007e-05 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.048004+0000", 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011082665 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:635 3.ds0 3:b9f64cc8:::smithi10536496-42:28 [read 0~3555328,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.841831020000001, 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014992814, 2024-09-15T03:42:00.984 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 635 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1600000000000002e-07 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036247+0000", 2024-09-15T03:42:00.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6290000000000002e-06 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036252+0000", 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2279999999999998e-06 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036256+0000", 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.303e-06 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.039208+0000", 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0029522020000000001 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.039315+0000", 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107315 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.051237+0000", 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011921721 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 78/49 ECSubWrite(tid=203, reqid=osd.13.0:203, at_version=78'56, trim_to=0'0, roll_forward_to=77'52))", 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:45.027703+0000", 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 15.850372029000001, 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.58712494400000004, 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:00.989 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027703+0000", 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027703+0000", 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027704+0000", 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027726+0000", 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1903999999999999e-05 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.613983+0000", 2024-09-15T03:42:00.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.58625706499999997 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.613990+0000", 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7920000000000004e-06 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614056+0000", 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6383999999999993e-05 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614066+0000", 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5889999999999992e-06 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614804+0000", 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073851799999999997 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614828+0000", 2024-09-15T03:42:00.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3951000000000002e-05 2024-09-15T03:42:00.994 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.994 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.994 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.994 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:00.994 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:00.994 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:00.995 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:00.994+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:00.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:42:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-09-15T03:42:01.017 INFO:tasks.rados.rados.0.smithi105.stdout:326: done (4 left) 2024-09-15T03:42:01.018 INFO:tasks.rados.rados.0.smithi105.stdout:327: delete oid 17 current snap is 29 2024-09-15T03:42:01.018 INFO:tasks.rados.rados.0.smithi105.stdout:323: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing write tid 2 to smithi10536496-13 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing write tid 3 to smithi10536496-13 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing write tid 4 to smithi10536496-13 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing write tid 5 to smithi10536496-13 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing write tid 6 to smithi10536496-13 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 51 (ObjNum 127 snap 29 seq_num 127) dirty exists 2024-09-15T03:42:01.025 INFO:tasks.rados.rados.0.smithi105.stdout:321: left oid 13 (ObjNum 127 snap 29 seq_num 127) 2024-09-15T03:42:01.026 INFO:tasks.rados.rados.0.smithi105.stdout:322: finishing copy_from racing read to smithi10536496-20 2024-09-15T03:42:01.026 INFO:tasks.rados.rados.0.smithi105.stdout:322: finishing copy_from to smithi10536496-20 2024-09-15T03:42:01.026 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 32 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-15T03:42:01.034 INFO:tasks.rados.rados.0.smithi105.stdout:321: done (4 left) 2024-09-15T03:42:01.034 INFO:tasks.rados.rados.0.smithi105.stdout:322: done (3 left) 2024-09-15T03:42:01.035 INFO:tasks.rados.rados.0.smithi105.stdout:323: done (2 left) 2024-09-15T03:42:01.035 INFO:tasks.rados.rados.0.smithi105.stdout:324: done (1 left) 2024-09-15T03:42:01.035 INFO:tasks.rados.rados.0.smithi105.stdout:327: done (0 left) 2024-09-15T03:42:01.035 INFO:tasks.rados.rados.0.smithi105.stdout:328: snap_remove snap 27 2024-09-15T03:42:01.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-15T03:42:01.102 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:01.102 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:01.102 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:42:01.103 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:01.111 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:42:01.121 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:01.120+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:01.197 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:01.195+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:01.250 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:42:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:42:01.250 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:01.257 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:42:01.297 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:01.296+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:01.355 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:53 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.318388419999998, 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.089580073999999996, 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.356 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 53 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035673+0000", 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035674+0000", 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7000000000000005e-07 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035677+0000", 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4589999999999998e-06 2024-09-15T03:42:01.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035684+0000", 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2659999999999998e-06 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035694+0000", 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8849999999999994e-06 2024-09-15T03:42:01.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036662+0000", 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096889100000000004 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.036718+0000", 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5436000000000003e-05 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037156+0000", 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043800399999999998 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037610+0000", 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00045398200000000002 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125179+0000", 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.087569006000000005 2024-09-15T03:42:01.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125253+0000", 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4574999999999997e-05 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:54 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.318362086, 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093566412000000002, 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 54 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.363 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035700+0000", 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035702+0000", 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1490000000000001e-06 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035709+0000", 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6630000000000004e-06 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.035713+0000", 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7780000000000001e-06 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037308+0000", 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015947769999999999 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037380+0000", 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2293999999999996e-05 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125568+0000", 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.088188213000000001 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125642+0000", 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4215999999999996e-05 2024-09-15T03:42:01.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.129266+0000", 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003623799 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:171 3.cs0 3:36fe1c1b:::smithi10536496-13:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.160075710999998, 2024-09-15T03:42:01.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62108922, 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 171 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193986+0000", 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193987+0000", 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3099999999999997e-07 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193989+0000", 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6410000000000002e-06 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193994+0000", 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6530000000000002e-06 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.193996+0000", 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2340000000000001e-06 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195359+0000", 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00136275 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195404+0000", 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5433999999999999e-05 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195711+0000", 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030663100000000003 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196156+0000", 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044516899999999998 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815021+0000", 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61886439199999999 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815075+0000", 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4685000000000002e-05 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:172 3.cs0 3:36fe1c1b:::smithi10536496-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.160044397999997, 2024-09-15T03:42:01.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.62217043699999997, 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 172 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194017+0000", 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194018+0000", 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5699999999999998e-07 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194020+0000", 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8729999999999999e-06 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194026+0000", 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1249999999999998e-06 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194028+0000", 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.475e-06 2024-09-15T03:42:01.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195860+0000", 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018314609999999999 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195941+0000", 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1464999999999993e-05 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815378+0000", 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61943704499999996 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815470+0000", 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1351e-05 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.816188+0000", 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071828500000000004 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:175 3.fs0 3:f17f8228:::smithi10536496-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.380 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.159998492, 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068787259000000003, 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 175 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:42:01.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194063+0000", 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194064+0000", 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2399999999999999e-07 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194065+0000", 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5540000000000001e-06 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194068+0000", 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0730000000000001e-06 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.194071+0000", 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4770000000000002e-06 2024-09-15T03:42:01.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195541+0000", 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014705670000000001 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.195649+0000", 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108057 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.262851+0000", 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067201307000000002 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 35/32 ECSubWriteReply(tid=30, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.157462170000002, 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61874190500000004, 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196600+0000", 2024-09-15T03:42:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.196603+0000", 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7719999999999999e-06 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814918+0000", 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.61831525899999995 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814925+0000", 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1629999999999999e-06 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.814972+0000", 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7231999999999998e-05 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.815342+0000", 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000369019 2024-09-15T03:42:01.388 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:300 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.349424999999997, 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0215413659999999, 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.389 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 300 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004637+0000", 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004638+0000", 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6020000000000001e-06 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004976+0000", 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033792499999999999 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.004994+0000", 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7507e-05 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.005001+0000", 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6499999999999996e-06 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008081+0000", 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003079675 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.008085+0000", 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3420000000000001e-06 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026131+0000", 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0180457119999999 2024-09-15T03:42:01.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026178+0000", 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6953000000000003e-05 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:301 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.344910741, 2024-09-15T03:42:01.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0170563379999999, 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 301 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009151+0000", 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009153+0000", 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4569999999999999e-06 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009921+0000", 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076796899999999997 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009955+0000", 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3696999999999999e-05 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009971+0000", 2024-09-15T03:42:01.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5441e-05 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010025+0000", 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4712000000000001e-05 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010029+0000", 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.63e-06 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026201+0000", 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016171661 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026207+0000", 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7710000000000001e-06 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.399 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:302 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.344075246000003, 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016248987, 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 302 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.009987+0000", 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7700000000000004e-07 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010036+0000", 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8857999999999999e-05 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010045+0000", 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6200000000000005e-06 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010049+0000", 2024-09-15T03:42:01.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3649999999999997e-06 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010226+0000", 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017731600000000001 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010228+0000", 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3090000000000001e-06 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026231+0000", 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016003078 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026236+0000", 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6639999999999997e-06 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.404 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:303 3.11s1 3.95fce6f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.344004890000001, 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.016202901, 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 303 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010057+0000", 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1399999999999997e-07 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010061+0000", 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010068+0000", 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9779999999999999e-06 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010072+0000", 2024-09-15T03:42:01.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0489999999999997e-06 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010289+0000", 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021628399999999999 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010290+0000", 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.201e-06 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026255+0000", 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015965209 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026260+0000", 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6659999999999999e-06 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.409 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:304 3.11s1 3.95fce6f1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.343986037000001, 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0162086930000001, 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 304 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010076+0000", 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9700000000000003e-07 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010078+0000", 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.711e-06 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010082+0000", 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1320000000000004e-06 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010084+0000", 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5450000000000002e-06 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010311+0000", 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022657600000000001 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010314+0000", 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.565e-06 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026281+0000", 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.015966967 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.026284+0000", 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8999999999999999e-06 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.414 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 47/32 ECSubWriteReply(tid=66, last_complete=47'18, committed=1, applied=1))", 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.341428796000002, 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16187408, 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012633+0000", 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3799999999999999e-07 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.012636+0000", 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.345e-06 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173941+0000", 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.16130529900000001 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.173963+0000", 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.243e-05 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174016+0000", 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2948000000000002e-05 2024-09-15T03:42:01.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.174507+0000", 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049081999999999999 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s0 3:cce4e8f8:::smithi10536496-4:head [create,setxattr Yt`Ot\\d[ZfNaQaqU[W\\PSa^pNWWUkki`hjlaoWcR (805) in=40b,setxattr ca[gmd`SV\\jp[aXclsea`kka[hPURW\\j[hao]pna (594) in=40b,setxattr jjQP\\pmOqcO\\iaNd[l\\^a]fXSRsgi\\sVYrQS[]ia (1614) in=40b,setxattr VeZgjt^aNniSnQhWPO[qm\\[itg[XR[[jXsbagns_ (454) in=40b,setxattr jath]ttaVem][bRrjP]pdOZWWo__SYO]VOaZhXRj (1858) in=40b,setxattr _oSdRYPilSVobVgWRbrP_piatjWaoYQ\\qcaVbh^g (138) in=40b,setxattr bTSZgV_k`ccTPieZXaVqe[OXkZefRqrsPPlisa^a (946) in=40b,setxattr OkXYkpcX_kP[TmZOosPhjlctOQRUagSbp`XaYrXW (910) in=40b,setxattr rjkSWecoRfpnaXdQmX`a`dPYt]P[_hRUfj`SdY[ (911) in=39b,setxattr OcknbT^]PQTVTj\\iodYl`oq[]Te]Us]e[nbQYY^N (1330) in=40b,setxattr r^qtXit^UbRgSb`o\\aoaSSnZqmZeWhhehTPZsgUO (418) in=40b,setxattr bnoZ`pV]YYNYqogeSaPofWR_PtahQZ_fR_h_nTnT (1190) in=40b,setxattr rjred]_aNiaaolkeaffPXTfZRd[[_PYZaUXNOUUt (33) in=40b,setxattr gnal\\XR[qfmfOecOmacYb_WWa\\W[rRPNWne`tlNT (1437) in=40b,setxattr sZmfXOTVQZ^VacNNaltSYOghkYNpdSfdZRhnXTn_ (4981) in=40b,setxattr TsTNdOVOpOPdoXYcpnamYadP^epnqllcbVsWNp^[ (17753) in=40b,setxattr PXkZNpT]nPaTrW]j`o_[RWiQrOrco]e_foQTXmO_ (1997) in=40b,setxattr sX[hspQmOVmae`epqt]pac_ZNQVn`YjUVU]QfSg^ (1051) in=40b,setxattr ^gV_afUghN\\gRs]pcaefnp[badg_a`]ROfd_ZZnW (5752) in=40b,setxattr d\\kRaqs`ZmPOe`_`ZimVtjOoQ[NieeSSZocTVNl_ (313) in=40b] snapc c=[c,b] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:42:01.418 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 45.342120528000002, 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0346458860000001, 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011941+0000", 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011942+0000", 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2100000000000002e-07 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011959+0000", 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7248000000000002e-05 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011965+0000", 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8880000000000002e-06 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.011969+0000", 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8929999999999998e-06 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.012000+0000", 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1004999999999999e-05 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.012002+0000", 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.289e-06 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.035078+0000", 2024-09-15T03:42:01.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0230762010000001 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.035294+0000", 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000215625 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.036366+0000", 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00107211 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046017+0000", 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096507329999999999 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046134+0000", 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011691199999999999 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046587+0000", 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000453561 2024-09-15T03:42:01.424 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:321 3.cs0 3:35272f51:::smithi10536496-18:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.270900957999999, 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11431013299999999, 2024-09-15T03:42:01.425 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 321 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083161+0000", 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083167+0000", 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5169999999999999e-06 2024-09-15T03:42:01.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083174+0000", 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9890000000000003e-06 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083178+0000", 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2220000000000004e-06 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.084760+0000", 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001582008 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.084856+0000", 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5461e-05 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085326+0000", 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046985000000000001 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.087826+0000", 2024-09-15T03:42:01.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025000759999999999 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197378+0000", 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.109552474 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197471+0000", 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2888999999999998e-05 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:322 3.cs0 3:35272f51:::smithi10536496-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:42:01.435 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.270876424999997, 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12070206999999999, 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 322 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083185+0000", 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083186+0000", 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.89e-07 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083188+0000", 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1009999999999999e-06 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083195+0000", 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2760000000000003e-06 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083199+0000", 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.41e-06 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085640+0000", 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002441838 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085722+0000", 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1780999999999995e-05 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197721+0000", 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.111998507 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197792+0000", 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0963000000000005e-05 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.203887+0000", 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060957049999999999 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 50/32 ECSubWriteReply(tid=75, last_complete=50'22, committed=1, applied=1))", 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.267098140000002, 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.110727595, 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086964+0000", 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5499999999999999e-07 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086965+0000", 2024-09-15T03:42:01.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6109999999999999e-06 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197067+0000", 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.110101613 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197072+0000", 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7940000000000002e-06 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197329+0000", 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025732199999999999 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197691+0000", 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036200000000000002 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.445 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 50/32 ECSubWriteReply(tid=80, last_complete=50'22, committed=1, applied=1))", 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.153456857999998, 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.82440747800000003, 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200605+0000", 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.91e-07 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.200607+0000", 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6500000000000001e-06 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024931+0000", 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.82432465200000005 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024938+0000", 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2129999999999996e-06 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.024985+0000", 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7713999999999998e-05 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:18.025012+0000", 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7058000000000001e-05 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:437 3.13s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:29.026636+0000", 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.327425345000002, 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.080767540999999998, 2024-09-15T03:42:01.449 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 437 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.026636+0000", 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.026636+0000", 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.026637+0000", 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6800000000000001e-07 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.026639+0000", 2024-09-15T03:42:01.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4889999999999998e-06 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.026647+0000", 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.026651+0000", 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5190000000000003e-06 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.029918+0000", 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0032671219999999999 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.030001+0000", 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2925999999999999e-05 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.036225+0000", 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062234150000000004 2024-09-15T03:42:01.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.036314+0000", 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9748999999999997e-05 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.107404+0000", 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071089543000000005 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s6 62/49 ECSubWrite(tid=177, reqid=osd.13.0:177, at_version=62'35, trim_to=0'0, roll_forward_to=58'33))", 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:29.107982+0000", 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.246079645000002, 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.237682268, 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.107982+0000", 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.107982+0000", 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.107983+0000", 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5599999999999995e-07 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.108007+0000", 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4552000000000001e-05 2024-09-15T03:42:01.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.344920+0000", 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.23691274600000001 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.344926+0000", 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6590000000000004e-06 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.344993+0000", 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7397000000000005e-05 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.345002+0000", 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9639999999999992e-06 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.345642+0000", 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000640451 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:29.345664+0000", 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1943e-05 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.13s0 70/69 ECSubReadReply(tid=133, attrs_read=0))", 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:37.881354+0000", 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.472707343, 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.123932283, 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.459 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:37.881354+0000", 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:37.881354+0000", 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:37.881356+0000", 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7239999999999999e-06 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:37.881798+0000", 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000441462 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.004959+0000", 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.123161061 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.004965+0000", 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0039999999999996e-06 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.005059+0000", 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4332000000000004e-05 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.005287+0000", 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022770000000000001 2024-09-15T03:42:01.462 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.463 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.463 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.463 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.463 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.463 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:01.465 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:42:01.465 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:01.463+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:01.563 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:01.563 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:01.563 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:42:01.564 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:01.565 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:01.564+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:01.572 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:42:01.665 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:01.664+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:01.670 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:01.670 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:01.670 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:42:01.670 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:42:01.670 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:01.679 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:42:01.767 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:01.765+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:01.776 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:42:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=14, to_read={3:0e027997:::smithi10536496-12:head=0,4096,0}, subchunks={3:0e027997:::smithi10536496-12:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.737172846999997, 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0095256449999999993, 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:42:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037684+0000", 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2799999999999996e-07 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.037696+0000", 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1408e-05 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046920+0000", 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092243440000000006 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046927+0000", 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8090000000000004e-06 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046974+0000", 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6869000000000001e-05 2024-09-15T03:42:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047209+0000", 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235687 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:51 3.5s0 3:ac8f3d24:::smithi10536496-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.736182446000001, 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01164283, 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 51 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038674+0000", 2024-09-15T03:42:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2599999999999998e-07 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038676+0000", 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.508e-06 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038680+0000", 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.653e-06 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.038685+0000", 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9269999999999996e-06 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040567+0000", 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00188288 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.040733+0000", 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016570300000000001 2024-09-15T03:42:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048135+0000", 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0074020889999999997 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048217+0000", 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2152999999999998e-05 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.050317+0000", 2024-09-15T03:42:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002099591 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=16, reqid=client.4562.0:65, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.727319743999999, 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020348845000000001, 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047537+0000", 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5899999999999996e-07 2024-09-15T03:42:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.047553+0000", 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6099999999999998e-05 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066726+0000", 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019172577 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066731+0000", 2024-09-15T03:42:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3340000000000001e-06 2024-09-15T03:42:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066780+0000", 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8708999999999997e-05 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.066790+0000", 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9709999999999999e-06 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067856+0000", 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001066259 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067886+0000", 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9335999999999998e-05 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:69 3.5s0 3:af19ecb4:::smithi10536496-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.726274099999998, 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094136830000000008, 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 69 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048582+0000", 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048583+0000", 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5900000000000002e-07 2024-09-15T03:42:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048585+0000", 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4729999999999999e-06 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048592+0000", 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2360000000000001e-06 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.048596+0000", 2024-09-15T03:42:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.092e-06 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051062+0000", 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024665440000000002 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.051144+0000", 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1369e-05 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056556+0000", 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054122759999999997 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.056628+0000", 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1699999999999995e-05 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057996+0000", 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001368534 2024-09-15T03:42:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.0s2 35/32 ECSubWrite(tid=18, reqid=client.4562.0:86, at_version=35'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.707434329999998, 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057601141000000002, 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067422+0000", 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067423+0000", 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2499999999999995e-07 2024-09-15T03:42:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.067437+0000", 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4445e-05 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121493+0000", 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054055923999999998 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121501+0000", 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8949999999999998e-06 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121585+0000", 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3502000000000001e-05 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121613+0000", 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8503000000000001e-05 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.124988+0000", 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033745590000000001 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.125023+0000", 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5787999999999997e-05 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:114 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.676898510000001, 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093237049999999998, 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 114 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097958+0000", 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1899999999999998e-07 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097960+0000", 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6130000000000001e-06 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097966+0000", 2024-09-15T03:42:01.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3690000000000003e-06 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.097970+0000", 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0069999999999999e-06 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.099933+0000", 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019621830000000002 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.100014+0000", 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0852000000000004e-05 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104698+0000", 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046846140000000001 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.104783+0000", 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4979000000000001e-05 2024-09-15T03:42:01.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.107282+0000", 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024986689999999998 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.0s2 35/32 ECSubRead(tid=19, to_read={3:0215fa29:::smithi10536496-22:head=0,4096,0}, subchunks={3:0215fa29:::smithi10536496-22:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.653436775000003, 2024-09-15T03:42:01.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039885337, 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121420+0000", 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8099999999999998e-07 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.121442+0000", 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2169999999999999e-05 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160985+0000", 2024-09-15T03:42:01.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039542246000000003 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160992+0000", 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2840000000000001e-06 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:42:01.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4248e-05 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161305+0000", 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025900800000000002 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.9s4 35/32 ECSubWrite(tid=25, reqid=client.4562.0:161, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:42:01.812 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.610801950000003, 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075462681000000004, 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164055+0000", 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9299999999999998e-07 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.164071+0000", 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5299e-05 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238920+0000", 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074849009999999994 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238927+0000", 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4039999999999998e-06 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238986+0000", 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9086000000000003e-05 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.238993+0000", 2024-09-15T03:42:01.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3719999999999999e-06 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239496+0000", 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050214699999999999 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.239517+0000", 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.177e-05 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:289 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.746799164999999, 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016317213000000001, 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 289 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.818 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028057+0000", 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028059+0000", 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3969999999999999e-06 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028066+0000", 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.627e-06 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028082+0000", 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5406000000000001e-05 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028087+0000", 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4809999999999997e-06 2024-09-15T03:42:01.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028181+0000", 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3881999999999993e-05 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028280+0000", 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9135999999999994e-05 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028913+0000", 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00063293699999999997 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.032481+0000", 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003567677 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044305+0000", 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011823828 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044375+0000", 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9842e-05 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:290 3.5s0 3:a7103f89:::smithi10536496-39:head [rollback b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.746756771000001, 2024-09-15T03:42:01.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016591980999999999, 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 290 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028100+0000", 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5999999999999999e-07 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028104+0000", 2024-09-15T03:42:01.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.501e-06 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028112+0000", 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2379999999999997e-06 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028116+0000", 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3290000000000004e-06 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029591+0000", 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001475004 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.029762+0000", 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000170557 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030258+0000", 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049562300000000005 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035019+0000", 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004761276 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044634+0000", 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096147730000000001 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.044692+0000", 2024-09-15T03:42:01.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8220000000000002e-05 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:291 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.746735602999998, 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019516250999999998, 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.830 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 291 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028121+0000", 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028123+0000", 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1239999999999999e-06 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028128+0000", 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5449999999999997e-06 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.028132+0000", 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0090000000000001e-06 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030541+0000", 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024089659999999998 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030601+0000", 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0384000000000003e-05 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.030921+0000", 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031995500000000002 2024-09-15T03:42:01.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.035034+0000", 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0041124630000000002 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047571+0000", 2024-09-15T03:42:01.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012537510999999999 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047637+0000", 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5925000000000003e-05 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=22, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:42:01.836 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.737864463999998, 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011072657, 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036992+0000", 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1899999999999999e-07 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.036994+0000", 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:42:01.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047471+0000", 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010477119 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047475+0000", 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3989999999999997e-06 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.047509+0000", 2024-09-15T03:42:01.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3426000000000002e-05 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.048065+0000", 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055612399999999996 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 45/44 ECSubWriteReply(tid=23, last_complete=45'13, committed=1, applied=1))", 2024-09-15T03:42:01.840 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.722093608999998, 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92706636600000003, 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:42:01.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052763+0000", 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9599999999999999e-07 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.052767+0000", 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8199999999999998e-06 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979592+0000", 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.92682499399999996 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979604+0000", 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1387e-05 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979726+0000", 2024-09-15T03:42:01.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122014 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:12.979829+0000", 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103655 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:313 3.5s0 3:a1c61493:::smithi10536496-38:head [append 0~147456] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:42:01.844 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.715898873, 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0098668620000000005, 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 313 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:42:01.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058958+0000", 2024-09-15T03:42:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.058959+0000", 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.108e-06 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059032+0000", 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3231000000000004e-05 2024-09-15T03:42:01.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059047+0000", 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4829e-05 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059053+0000", 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5910000000000001e-06 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.060287+0000", 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012338760000000001 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.060443+0000", 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015520099999999999 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.061055+0000", 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00061251899999999995 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068673+0000", 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0076178460000000002 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068754+0000", 2024-09-15T03:42:01.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1104000000000001e-05 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068825+0000", 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0556999999999995e-05 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:314 3.5s0 3:a1c61493:::smithi10536496-38:head [setxattr _header (56) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.715795780999997, 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015366338, 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 314 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059061+0000", 2024-09-15T03:42:01.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-07 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059066+0000", 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8790000000000002e-06 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059075+0000", 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6009999999999995e-06 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059079+0000", 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999997e-06 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.063568+0000", 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044885849999999998 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.063652+0000", 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4670000000000001e-05 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.064159+0000", 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050645400000000004 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.068840+0000", 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046815729999999996 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074353+0000", 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055131449999999997 2024-09-15T03:42:01.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074427+0000", 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3767e-05 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:315 3.5s0 3:a1c61493:::smithi10536496-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.715773106999997, 2024-09-15T03:42:01.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018255761999999998, 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 315 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059083+0000", 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7999999999999996e-07 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059086+0000", 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1390000000000002e-06 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3779999999999998e-06 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.059098+0000", 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6910000000000002e-06 2024-09-15T03:42:01.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.065037+0000", 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005938484 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:01.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.065126+0000", 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9343000000000002e-05 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074804+0000", 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096778360000000004 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.074926+0000", 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122092 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.077339+0000", 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024133190000000001 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:325 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.691847111999998, 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.078787472999999997, 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:01.864 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 325 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083009+0000", 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083010+0000", 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.46e-07 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083012+0000", 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4679999999999999e-06 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083020+0000", 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5769999999999996e-06 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.083024+0000", 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.143e-06 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085552+0000", 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025276999999999999 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.085630+0000", 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7831999999999996e-05 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.086020+0000", 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039002200000000001 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.090715+0000", 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046956599999999999 2024-09-15T03:42:01.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.161716+0000", 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071000895999999994 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.161797+0000", 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0729000000000006e-05 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.11s2 71/49 ECSubWrite(tid=27, reqid=client.4562.0:542, at_version=71'31, trim_to=0'0, roll_forward_to=71'27))", 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.115321+0000", 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.659535352999999, 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.83408876300000001, 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.115321+0000", 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.115321+0000", 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.115322+0000", 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.115334+0000", 2024-09-15T03:42:01.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1987999999999999e-05 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.948916+0000", 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.83358260299999998 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.948920+0000", 2024-09-15T03:42:01.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1420000000000003e-06 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.948973+0000", 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2423000000000003e-05 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.948979+0000", 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2029999999999997e-06 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.949393+0000", 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000414318 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.949410+0000", 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6623999999999999e-05 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.875 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.8s5 77/44 ECSubWrite(tid=96, reqid=client.4562.0:631, at_version=77'46, trim_to=0'0, roll_forward_to=77'45))", 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:44.058359+0000", 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.716497168, 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.44048110200000001, 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.058359+0000", 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.058359+0000", 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.058360+0000", 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3500000000000005e-07 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.060383+0000", 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020226689999999999 2024-09-15T03:42:01.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.496922+0000", 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.43653869899999997 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.496927+0000", 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7209999999999999e-06 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.497003+0000", 2024-09-15T03:42:01.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5742999999999996e-05 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.497015+0000", 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2300000000000001e-05 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.498808+0000", 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017930050000000001 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.498841+0000", 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2129999999999999e-05 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.13s5 83/82 ECSubWrite(tid=172, reqid=osd.11.0:172, at_version=83'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:50.027755+0000", 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 11.747101358, 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.15104046900000001, 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:01.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.027755+0000", 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.027755+0000", 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.027756+0000", 2024-09-15T03:42:01.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0699999999999996e-07 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.027778+0000", 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1922000000000001e-05 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.177916+0000", 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.15013842399999999 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.177920+0000", 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.155e-06 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.177977+0000", 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6631e-05 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.177983+0000", 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0159999999999997e-06 2024-09-15T03:42:01.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.178776+0000", 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079314100000000003 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:50.178796+0000", 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9473000000000001e-05 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:01.886 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:01.886 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:01.887 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:42:01.888 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:01.886+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:01.988 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:01.987+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:02.016 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-09-15T03:42:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:328: done (0 left) 2024-09-15T03:42:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:329: copy_from oid 17 from oid 31 current snap is 29 2024-09-15T03:42:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 39 current snap is 29 2024-09-15T03:42:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:330: seq_num 129 ranges {245760=622592,868352=655360,1523712=688128,2211840=212992} 2024-09-15T03:42:02.022 INFO:tasks.rados.rados.0.smithi105.stdout:330: writing smithi10536496-39 from 245760 to 868352 tid 1 2024-09-15T03:42:02.027 INFO:tasks.rados.rados.0.smithi105.stdout:330: writing smithi10536496-39 from 868352 to 1523712 tid 2 2024-09-15T03:42:02.030 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:02.030 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:02.030 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:42:02.030 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:02.031 INFO:tasks.rados.rados.0.smithi105.stdout:330: writing smithi10536496-39 from 1523712 to 2211840 tid 3 2024-09-15T03:42:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:330: writing smithi10536496-39 from 2211840 to 2424832 tid 4 2024-09-15T03:42:02.032 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 12 current snap is 29 2024-09-15T03:42:02.033 INFO:tasks.rados.rados.0.smithi105.stdout:331: seq_num 130 ranges {1785856=638976,2424832=671744,3096576=802816,3899392=81920} 2024-09-15T03:42:02.039 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-15T03:42:02.041 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:42:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:331: writing smithi10536496-12 from 1785856 to 2424832 tid 1 2024-09-15T03:42:02.050 INFO:tasks.rados.rados.0.smithi105.stdout:331: writing smithi10536496-12 from 2424832 to 3096576 tid 2 2024-09-15T03:42:02.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:42:02.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:42:02.054 INFO:tasks.rados.rados.0.smithi105.stdout:331: writing smithi10536496-12 from 3096576 to 3899392 tid 3 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:331: writing smithi10536496-12 from 3899392 to 3981312 tid 4 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:332: snap_create 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:329: finishing copy_from racing read to smithi10536496-17 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing write tid 1 to smithi10536496-39 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing write tid 2 to smithi10536496-39 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing write tid 3 to smithi10536496-39 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:329: finishing copy_from to smithi10536496-17 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 32 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-15T03:42:02.055 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing write tid 1 to smithi10536496-12 2024-09-15T03:42:02.058 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing write tid 2 to smithi10536496-12 2024-09-15T03:42:02.059 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing write tid 3 to smithi10536496-12 2024-09-15T03:42:02.059 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing write tid 4 to smithi10536496-12 2024-09-15T03:42:02.061 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing write tid 5 to smithi10536496-12 2024-09-15T03:42:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing write tid 6 to smithi10536496-12 2024-09-15T03:42:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 69 (ObjNum 130 snap 29 seq_num 130) dirty exists 2024-09-15T03:42:02.064 INFO:tasks.rados.rados.0.smithi105.stdout:331: left oid 12 (ObjNum 130 snap 29 seq_num 130) 2024-09-15T03:42:02.126 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:02.125+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:02.139 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:02.139 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:02.139 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:42:02.139 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:42:02.139 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:02.147 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:42:02.189 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:02.188+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:02.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:42:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:42:02.237 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:42:02.237 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:42:02.257 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:42:02.257 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:42:02.277 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:42:02.277 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:42:02.286 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:02.286 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:02.290 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:42:02.289+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:42:02.294 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:42:02.297 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:42:02.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:42:02.307 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing write tid 4 to smithi10536496-39 2024-09-15T03:42:02.309 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing write tid 5 to smithi10536496-39 2024-09-15T03:42:02.333 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:42:02.333 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:42:02.353 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:42:02.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:42:02.390 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:42:02.390 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:42:02.391 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:42:02.390+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:42:02.400 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:02.400 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:02.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:42:02.410 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:42:02.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:42:02.448 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:42:02.448 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:42:02.468 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:42:02.468 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:42:02.469 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing write tid 6 to smithi10536496-39 2024-09-15T03:42:02.469 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 56 (ObjNum 129 snap 29 seq_num 129) dirty exists 2024-09-15T03:42:02.469 INFO:tasks.rados.rados.0.smithi105.stdout:330: left oid 39 (ObjNum 129 snap 29 seq_num 129) 2024-09-15T03:42:02.487 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:42:02.488 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:42:02.492 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:02.491+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:02.507 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:02.507 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:02.512 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:02.513 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:02.522 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:42:02.527 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:02.527 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:42:02.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:02.545+0000 7fa7e1026640 0 -- 172.21.15.5:0/2912217250 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7fa7dc195090 msgr2=0x7fa7dc1974f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:02.553 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:02.574 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:02.574 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:02.593 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:02.592+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:02.630 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:02.630 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:02.638 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:42:02.694 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:02.693+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:02.740 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:02.741 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:02.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:42:02.833 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:02.832+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:02.891 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:02.896 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:02.895+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:02.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:42:02.997 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:02.997+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:03.041 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.042 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:42:03.051 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-09-15T03:42:03.051 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:42:03.052 INFO:tasks.rados.rados.0.smithi105.stdout:329: done (3 left) 2024-09-15T03:42:03.052 INFO:tasks.rados.rados.0.smithi105.stdout:330: done (2 left) 2024-09-15T03:42:03.052 INFO:tasks.rados.rados.0.smithi105.stdout:331: done (1 left) 2024-09-15T03:42:03.052 INFO:tasks.rados.rados.0.smithi105.stdout:332: done (0 left) 2024-09-15T03:42:03.052 INFO:tasks.rados.rados.0.smithi105.stdout:333: setattr oid 1 current snap is 30 2024-09-15T03:42:03.054 INFO:tasks.rados.rados.0.smithi105.stdout:334: snap_create 2024-09-15T03:42:03.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-15T03:42:03.098 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:03.097+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.156 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:42:03.200 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:42:03.199+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:42:03.270 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.271 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.279 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:42:03.301 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:03.300+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:03.381 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.381 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.390 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:42:03.402 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:03.401+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:03.491 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.492 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.500 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:42:03.503 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:03.502+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:03.601 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.602 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.604 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:03.603+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:03.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:42:03.705 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:42:03.704+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:42:03.711 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.711 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:42:03.805 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:03.805+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:03.835 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.835 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:42:03.907 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:03.906+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:03.988 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:03.988 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:03.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:42:04.008 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:04.007+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:04.010 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-09-15T03:42:04.012 INFO:tasks.rados.rados.0.smithi105.stdout:334: done (1 left) 2024-09-15T03:42:04.012 INFO:tasks.rados.rados.0.smithi105.stdout:335: copy_from oid 11 from oid 5 current snap is 31 2024-09-15T03:42:04.012 INFO:tasks.rados.rados.0.smithi105.stdout:336: read oid 15 snap -1 2024-09-15T03:42:04.012 INFO:tasks.rados.rados.0.smithi105.stdout:336: expect (ObjNum 87 snap 14 seq_num 87) 2024-09-15T03:42:04.012 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 33 current snap is 31 2024-09-15T03:42:04.012 INFO:tasks.rados.rados.0.smithi105.stdout:337: seq_num 131 ranges {540672=655360,1196032=425984,1622016=638976,2260992=491520} 2024-09-15T03:42:04.018 INFO:tasks.rados.rados.0.smithi105.stdout:337: writing smithi10536496-33 from 540672 to 1196032 tid 1 2024-09-15T03:42:04.021 INFO:tasks.rados.rados.0.smithi105.stdout:337: writing smithi10536496-33 from 1196032 to 1622016 tid 2 2024-09-15T03:42:04.025 INFO:tasks.rados.rados.0.smithi105.stdout:337: writing smithi10536496-33 from 1622016 to 2260992 tid 3 2024-09-15T03:42:04.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-15T03:42:04.029 INFO:tasks.rados.rados.0.smithi105.stdout:337: writing smithi10536496-33 from 2260992 to 2752512 tid 4 2024-09-15T03:42:04.029 INFO:tasks.rados.rados.0.smithi105.stdout:338: snap_create 2024-09-15T03:42:04.030 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing write tid 1 to smithi10536496-33 2024-09-15T03:42:04.037 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing write tid 2 to smithi10536496-33 2024-09-15T03:42:04.037 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing write tid 3 to smithi10536496-33 2024-09-15T03:42:04.038 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing write tid 4 to smithi10536496-33 2024-09-15T03:42:04.038 INFO:tasks.rados.rados.0.smithi105.stdout:336: expect (ObjNum 87 snap 14 seq_num 87) 2024-09-15T03:42:04.104 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.104 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.108 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:04.108+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:04.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:42:04.151 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 76 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T03:42:04.210 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:04.208+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:04.215 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.215 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:42:04.311 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:04.310+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:04.325 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.325 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:42:04.412 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:04.411+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:04.431 INFO:tasks.rados.rados.0.smithi105.stdout:335: finishing copy_from racing read to smithi10536496-11 2024-09-15T03:42:04.435 INFO:tasks.rados.rados.0.smithi105.stdout:335: finishing copy_from to smithi10536496-11 2024-09-15T03:42:04.435 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-15T03:42:04.436 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.436 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:42:04.513 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:04.512+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:04.548 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.548 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:42:04.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:42:04.614 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:04.613+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:04.659 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.659 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:42:04.700 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing write tid 5 to smithi10536496-33 2024-09-15T03:42:04.701 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing write tid 6 to smithi10536496-33 2024-09-15T03:42:04.701 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 89 (ObjNum 131 snap 31 seq_num 131) dirty exists 2024-09-15T03:42:04.701 INFO:tasks.rados.rados.0.smithi105.stdout:337: left oid 33 (ObjNum 131 snap 31 seq_num 131) 2024-09-15T03:42:04.715 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:04.715+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:04.715 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:42:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:42:04.737 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:42:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:42:04.757 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:42:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:42:04.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:42:04.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:42:04.796 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:42:04.796 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:42:04.808 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.809 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.817 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:42:04.818 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:42:04.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:42:04.819 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:04.815+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:04.837 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:42:04.837 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:42:04.857 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:42:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:42:04.876 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:42:04.876 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:42:04.897 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:42:04.897 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:42:04.917 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:04.917+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:04.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:42:04.918 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:42:04.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:42:04.938 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:42:04.957 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:04.958 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:04.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:04.956+0000 7f535c939640 0 -- 172.21.15.5:0/3963036406 >> [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] conn(0x7f53581924f0 msgr2=0x7f5358194950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:04.962 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:04.962 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:04.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:42:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-09-15T03:42:05.014 INFO:tasks.rados.rados.0.smithi105.stdout:333: done (4 left) 2024-09-15T03:42:05.014 INFO:tasks.rados.rados.0.smithi105.stdout:335: done (3 left) 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:336: done (2 left) 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:337: done (1 left) 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:338: done (0 left) 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:339: read oid 6 snap 31 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:339: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:340: rollback oid 37 current snap is 32 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 37 to 30 2024-09-15T03:42:05.015 INFO:tasks.rados.rados.0.smithi105.stdout:341: setattr oid 21 current snap is 32 2024-09-15T03:42:05.017 INFO:tasks.rados.rados.0.smithi105.stdout:342: read oid 3 snap 29 2024-09-15T03:42:05.018 INFO:tasks.rados.rados.0.smithi105.stdout:342: expect (ObjNum 109 snap 17 seq_num 109) 2024-09-15T03:42:05.018 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 26 current snap is 32 2024-09-15T03:42:05.018 INFO:tasks.rados.rados.0.smithi105.stdout:343: seq_num 132 ranges {0=671744,671744=573440,1245184=753664,1998848=229376} 2024-09-15T03:42:05.021 INFO:tasks.rados.rados.0.smithi105.stdout:343: writing smithi10536496-26 from 0 to 671744 tid 1 2024-09-15T03:42:05.021 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:05.020+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:05.025 INFO:tasks.rados.rados.0.smithi105.stdout:343: writing smithi10536496-26 from 671744 to 1245184 tid 2 2024-09-15T03:42:05.029 INFO:tasks.rados.rados.0.smithi105.stdout:343: writing smithi10536496-26 from 1245184 to 1998848 tid 3 2024-09-15T03:42:05.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-15T03:42:05.031 INFO:tasks.rados.rados.0.smithi105.stdout:343: writing smithi10536496-26 from 1998848 to 2228224 tid 4 2024-09-15T03:42:05.031 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 36 current snap is 32 2024-09-15T03:42:05.031 INFO:tasks.rados.rados.0.smithi105.stdout:344: seq_num 133 ranges {2342912=688128,3031040=720896,3751936=475136,4227072=360448} 2024-09-15T03:42:05.048 INFO:tasks.rados.rados.0.smithi105.stdout:344: writing smithi10536496-36 from 2342912 to 3031040 tid 1 2024-09-15T03:42:05.052 INFO:tasks.rados.rados.0.smithi105.stdout:344: writing smithi10536496-36 from 3031040 to 3751936 tid 2 2024-09-15T03:42:05.055 INFO:tasks.rados.rados.0.smithi105.stdout:344: writing smithi10536496-36 from 3751936 to 4227072 tid 3 2024-09-15T03:42:05.057 INFO:tasks.rados.rados.0.smithi105.stdout:344: writing smithi10536496-36 from 4227072 to 4587520 tid 4 2024-09-15T03:42:05.057 INFO:tasks.rados.rados.0.smithi105.stdout:345: delete oid 9 current snap is 32 2024-09-15T03:42:05.057 INFO:tasks.rados.rados.0.smithi105.stdout:339: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-15T03:42:05.060 INFO:tasks.rados.rados.0.smithi105.stdout:340: finishing rollback tid 0 to smithi10536496-37 2024-09-15T03:42:05.060 INFO:tasks.rados.rados.0.smithi105.stdout:340: finishing rollback tid 1 to smithi10536496-37 2024-09-15T03:42:05.060 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 78 (ObjNum 124 snap 27 seq_num 124) dirty exists 2024-09-15T03:42:05.060 INFO:tasks.rados.rados.0.smithi105.stdout:340: finishing rollback tid 2 to smithi10536496-37 2024-09-15T03:42:05.060 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 70 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-15T03:42:05.061 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing write tid 1 to smithi10536496-26 2024-09-15T03:42:05.061 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing write tid 2 to smithi10536496-26 2024-09-15T03:42:05.061 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing write tid 3 to smithi10536496-26 2024-09-15T03:42:05.061 INFO:tasks.rados.rados.0.smithi105.stdout:342: expect (ObjNum 109 snap 17 seq_num 109) 2024-09-15T03:42:05.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:42:05.084 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:05.085 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:05.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:42:05.120 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:05.119+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:05.150 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing write tid 4 to smithi10536496-26 2024-09-15T03:42:05.150 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing write tid 1 to smithi10536496-36 2024-09-15T03:42:05.150 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing write tid 2 to smithi10536496-36 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing write tid 3 to smithi10536496-36 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing write tid 4 to smithi10536496-36 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:339: done (6 left) 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:340: done (5 left) 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:341: done (4 left) 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:342: done (3 left) 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:345: done (2 left) 2024-09-15T03:42:05.151 INFO:tasks.rados.rados.0.smithi105.stdout:346: delete oid 7 current snap is 32 2024-09-15T03:42:05.153 INFO:tasks.rados.rados.0.smithi105.stdout:346: done (2 left) 2024-09-15T03:42:05.153 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 17 current snap is 32 2024-09-15T03:42:05.154 INFO:tasks.rados.rados.0.smithi105.stdout:347: seq_num 136 ranges {1736704=737280,2473984=770048,3244032=540672,3784704=245760} 2024-09-15T03:42:05.166 INFO:tasks.rados.rados.0.smithi105.stdout:347: writing smithi10536496-17 from 1736704 to 2473984 tid 1 2024-09-15T03:42:05.171 INFO:tasks.rados.rados.0.smithi105.stdout:347: writing smithi10536496-17 from 2473984 to 3244032 tid 2 2024-09-15T03:42:05.174 INFO:tasks.rados.rados.0.smithi105.stdout:347: writing smithi10536496-17 from 3244032 to 3784704 tid 3 2024-09-15T03:42:05.175 INFO:tasks.rados.rados.0.smithi105.stdout:347: writing smithi10536496-17 from 3784704 to 4030464 tid 4 2024-09-15T03:42:05.175 INFO:tasks.rados.rados.0.smithi105.stdout:348: setattr oid 13 current snap is 32 2024-09-15T03:42:05.175 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing write tid 1 to smithi10536496-17 2024-09-15T03:42:05.176 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing write tid 2 to smithi10536496-17 2024-09-15T03:42:05.178 INFO:tasks.rados.rados.0.smithi105.stdout:349: read oid 30 snap -1 2024-09-15T03:42:05.179 INFO:tasks.rados.rados.0.smithi105.stdout:349: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:42:05.179 INFO:tasks.rados.rados.0.smithi105.stdout:350: snap_create 2024-09-15T03:42:05.179 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing write tid 3 to smithi10536496-17 2024-09-15T03:42:05.179 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing write tid 4 to smithi10536496-17 2024-09-15T03:42:05.180 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing write tid 5 to smithi10536496-17 2024-09-15T03:42:05.182 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 55 (ObjNum 127 snap 29 seq_num 127) dirty exists 2024-09-15T03:42:05.182 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing write tid 6 to smithi10536496-17 2024-09-15T03:42:05.182 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 45 (ObjNum 136 snap 32 seq_num 136) dirty exists 2024-09-15T03:42:05.183 INFO:tasks.rados.rados.0.smithi105.stdout:347: left oid 17 (ObjNum 136 snap 32 seq_num 136) 2024-09-15T03:42:05.183 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:05.183 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:42:05.189 INFO:tasks.rados.rados.0.smithi105.stdout:349: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:42:05.204 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:05.205 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:42:05.221 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:05.220+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:05.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:05.225 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:05.243 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:42:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 81.392259526000004, 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:42:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:42:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:42:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:42:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:42:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 81.392231421000005, 2024-09-15T03:42:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:42:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:42:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:42:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 81.097390314999998, 2024-09-15T03:42:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:42:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:42:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:42:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=8, to_read={3:fff44eab:::smithi10536496-10:head=0,4096,0}, subchunks={3:fff44eab:::smithi10536496-10:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.207813971, 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046423195, 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033900+0000", 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033906+0000", 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080075+0000", 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046169038000000003 2024-09-15T03:42:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080080+0000", 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080125+0000", 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5151000000000002e-05 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080322+0000", 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019686500000000001 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 35/32 ECSubWriteReply(tid=118, last_complete=35'2, committed=1, applied=1))", 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.154881689, 2024-09-15T03:42:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021508349, 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:42:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086832+0000", 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086835+0000", 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0479999999999999e-06 2024-09-15T03:42:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108275+0000", 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021439848000000001 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108283+0000", 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8609999999999998e-06 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108320+0000", 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7330999999999999e-05 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108340+0000", 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9531999999999999e-05 2024-09-15T03:42:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.133308323000001, 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041863826999999999, 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108419+0000", 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149106+0000", 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040686870999999999 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149110+0000", 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.335e-06 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149157+0000", 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7131e-05 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149164+0000", 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150245+0000", 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010804580000000001 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150269+0000", 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4128e-05 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:132 3.3s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.116498946, 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015465478, 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 132 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125215+0000", 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125216+0000", 2024-09-15T03:42:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7349999999999999e-06 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125223+0000", 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125227+0000", 2024-09-15T03:42:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8840000000000004e-06 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129903+0000", 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046767349999999996 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129995+0000", 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1706000000000006e-05 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133187+0000", 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031916510000000002 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133234+0000", 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7862999999999999e-05 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140680+0000", 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074451860000000003 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=25, to_read={3:fc38c57e:::smithi10536496-49:head=0,4096,0}, subchunks={3:fc38c57e:::smithi10536496-49:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.085258183999997, 2024-09-15T03:42:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038157982, 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:42:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156460+0000", 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194316+0000", 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037855609999999998 2024-09-15T03:42:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194320+0000", 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999998e-06 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194361+0000", 2024-09-15T03:42:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0708999999999999e-05 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194613+0000", 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025227700000000002 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=31, to_read={3:f17f8228:::smithi10536496-40:head=0,167936,0}, subchunks={3:f17f8228:::smithi10536496-40:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.045809994000003, 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066290004, 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:42:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195904+0000", 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195911+0000", 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3030000000000002e-06 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262021+0000", 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066109629000000003 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262025+0000", 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5909999999999998e-06 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262085+0000", 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0183999999999998e-05 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262193+0000", 2024-09-15T03:42:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107648 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:277 3.13s0 3:ca2bd2fa:::smithi10536496-45:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.212158956000003, 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0157117019999999, 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 277 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029556+0000", 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.542e-06 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029563+0000", 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2989999999999999e-06 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029577+0000", 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029601+0000", 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3200000000000001e-05 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029661+0000", 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0633e-05 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029665+0000", 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5990000000000002e-06 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037509+0000", 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0078444609999999 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037649+0000", 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139476 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.045266+0000", 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076171729999999997 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.5s3 47/44 ECSubWrite(tid=24, reqid=osd.15.0:24, at_version=47'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-15T03:42:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.238537315000002, 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013250879, 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003176+0000", 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4600000000000001e-07 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.003206+0000", 2024-09-15T03:42:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9589000000000001e-05 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004002+0000", 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079689099999999998 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004013+0000", 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0513000000000001e-05 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004302+0000", 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000288554 2024-09-15T03:42:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.004314+0000", 2024-09-15T03:42:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2277e-05 2024-09-15T03:42:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016390+0000", 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012076 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016427+0000", 2024-09-15T03:42:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6909000000000002e-05 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:305 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.237221081000001, 2024-09-15T03:42:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0410521429999999, 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 305 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004494+0000", 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1660000000000001e-06 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004502+0000", 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1540000000000002e-06 2024-09-15T03:42:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004520+0000", 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7742000000000001e-05 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004530+0000", 2024-09-15T03:42:05.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004681+0000", 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015092999999999999 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004692+0000", 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1232e-05 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009082+0000", 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043904649999999996 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009087+0000", 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7339999999999999e-06 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.033828+0000", 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0247413160000001 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.034013+0000", 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018443999999999999 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045544+0000", 2024-09-15T03:42:05.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011531351 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:306 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.237124901999998, 2024-09-15T03:42:05.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0413215819999999, 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 306 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004590+0000", 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3260000000000001e-06 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004596+0000", 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010000000000002e-06 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004613+0000", 2024-09-15T03:42:05.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6892999999999999e-05 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004623+0000", 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268000000000001e-05 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009818+0000", 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051952880000000002 2024-09-15T03:42:05.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009821+0000", 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0189999999999998e-06 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035613+0000", 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.025791608 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035700+0000", 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7298000000000003e-05 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045910+0000", 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010209480999999999 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:307 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.237080988000002, 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.041622225, 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 307 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004633+0000", 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0200000000000001e-07 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004639+0000", 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5599999999999999e-06 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004649+0000", 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004656+0000", 2024-09-15T03:42:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8870000000000003e-06 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051891749999999999 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035995+0000", 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026149322 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.036088+0000", 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3162e-05 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.046254+0000", 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010166405999999999 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:335 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.120912949999997, 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077783689000000003, 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 335 2024-09-15T03:42:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120802+0000", 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:42:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120808+0000", 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2999999999999998e-06 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120821+0000", 2024-09-15T03:42:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120826+0000", 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7879999999999997e-06 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120884+0000", 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8034e-05 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.121005+0000", 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120637 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.130702+0000", 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096976079999999999 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.132697+0000", 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019945800000000001 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198536+0000", 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065839133999999994 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198584+0000", 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7796000000000001e-05 2024-09-15T03:42:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:369 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~589824] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.216434571999997, 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022704654000000001, 2024-09-15T03:42:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 369 2024-09-15T03:42:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025281+0000", 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4159999999999998e-06 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028105+0000", 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028242380000000002 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028126+0000", 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0939999999999999e-05 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028134+0000", 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7649999999999993e-06 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032973+0000", 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048395670000000003 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.033153+0000", 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017928700000000001 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034716+0000", 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001562873 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047854+0000", 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013138212999999999 2024-09-15T03:42:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047923+0000", 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9306000000000004e-05 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047983+0000", 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0049000000000001e-05 2024-09-15T03:42:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:370 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~573440] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.208749273999999, 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018633602999999999, 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 370 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032966+0000", 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034471+0000", 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015050790000000001 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034522+0000", 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0529999999999999e-05 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034528+0000", 2024-09-15T03:42:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1750000000000002e-06 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041526+0000", 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006997886 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041652+0000", 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126415 2024-09-15T03:42:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.042320+0000", 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066831300000000002 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051454+0000", 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091331659999999999 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051523+0000", 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9029999999999995e-05 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051597+0000", 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4819000000000003e-05 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:371 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~196608] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.207175132000003, 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827797000000001, 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 371 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:42:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034539+0000", 2024-09-15T03:42:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.116e-06 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034715+0000", 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017624599999999999 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034727+0000", 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1912999999999999e-05 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034736+0000", 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044278+0000", 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095422289999999993 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044402+0000", 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124227 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.045119+0000", 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071647099999999999 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054221+0000", 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091019350000000002 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054291+0000", 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0148999999999995e-05 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054366+0000", 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4883999999999997e-05 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:372 3.18s0 3:1c642a04:::smithi10536496-46:head [setxattr _header (56) in=7b] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.206968617999998, 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020478778, 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 372 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4199999999999995e-07 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034749+0000", 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034759+0000", 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4409999999999999e-06 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034769+0000", 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0064999999999999e-05 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047175+0000", 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012405657 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047305+0000", 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013068800000000001 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047652+0000", 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034670499999999999 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054413+0000", 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067614540000000001 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055174+0000", 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076077999999999996 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055223+0000", 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9076999999999998e-05 2024-09-15T03:42:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:373 3.18s0 3:1c642a04:::smithi10536496-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.206938690000001, 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021752218, 2024-09-15T03:42:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 373 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:42:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034775+0000", 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034778+0000", 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9249999999999999e-06 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034785+0000", 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5669999999999997e-06 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034790+0000", 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.51e-06 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048073+0000", 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013283497 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048128+0000", 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4455e-05 2024-09-15T03:42:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055532+0000", 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074043889999999999 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055612+0000", 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9971999999999999e-05 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.056527+0000", 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091438399999999997 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:05.385 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:05.387 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:42:05.387 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:05.386+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:05.488 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:05.487+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:05.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:05.490 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:42:05.490 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:05.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:42:05.589 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:05.589+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:05.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:42:05.597 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:42:05.597 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:05.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:42:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing write tid 5 to smithi10536496-36 2024-09-15T03:42:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing write tid 6 to smithi10536496-36 2024-09-15T03:42:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 84 (ObjNum 133 snap 32 seq_num 133) dirty exists 2024-09-15T03:42:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:344: left oid 36 (ObjNum 133 snap 32 seq_num 133) 2024-09-15T03:42:05.690 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:05.690+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:05.705 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:6 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.686280506999999, 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012850047, 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 6 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016665+0000", 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6100000000000001e-07 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016667+0000", 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9769999999999999e-06 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016674+0000", 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.016678+0000", 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2100000000000003e-06 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018325+0000", 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001646385 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.018401+0000", 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6032999999999995e-05 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027234+0000", 2024-09-15T03:42:05.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088331369999999996 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.027316+0000", 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1862999999999999e-05 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.029515+0000", 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002199281 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:60 3.es0 3:7dd49206:::smithi10536496-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.662273415000001, 2024-09-15T03:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010867740000000001, 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 60 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040672+0000", 2024-09-15T03:42:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040673+0000", 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2399999999999999e-07 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040675+0000", 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2479999999999999e-06 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040681+0000", 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0170000000000002e-06 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.040689+0000", 2024-09-15T03:42:05.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6330000000000002e-06 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042594+0000", 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019041489999999999 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.042676+0000", 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2633999999999995e-05 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049349+0000", 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066727380000000001 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.049419+0000", 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9868000000000003e-05 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.051540+0000", 2024-09-15T03:42:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002121029 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:84 3.es0 3:7f98869d:::smithi10536496-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.640268282000001, 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011714999, 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 84 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062677+0000", 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062678+0000", 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062679+0000", 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6920000000000001e-06 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062685+0000", 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0979999999999999e-06 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.062688+0000", 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3860000000000001e-06 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064229+0000", 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015411999999999999 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.064308+0000", 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8621000000000005e-05 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070977+0000", 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006669538 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.071049+0000", 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1464000000000006e-05 2024-09-15T03:42:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.074392+0000", 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003343586 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:152 3.es0 3:71615b7a:::smithi10536496-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:42:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.547333242000001, 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.031528529, 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 152 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:42:05.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155612+0000", 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155613+0000", 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9599999999999999e-07 2024-09-15T03:42:05.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155616+0000", 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.88e-06 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155623+0000", 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6950000000000003e-06 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155627+0000", 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999997e-06 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156746+0000", 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001119229 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.156828+0000", 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2294999999999997e-05 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157113+0000", 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028439999999999997 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.159322+0000", 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022090719999999999 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:05.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187068+0000", 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.027746694999999998 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187141+0000", 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:153 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:42:05.730 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.547311892000003, 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.032862542000000002, 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 153 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:42:05.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155634+0000", 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8999999999999997e-07 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155636+0000", 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7260000000000001e-06 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155642+0000", 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1449999999999996e-06 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.155645+0000", 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4470000000000002e-06 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157320+0000", 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016747769999999999 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.157366+0000", 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5982999999999997e-05 2024-09-15T03:42:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187194+0000", 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.029827955 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187240+0000", 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5494000000000002e-05 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188496+0000", 2024-09-15T03:42:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012565250000000001 2024-09-15T03:42:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:166 3.es0 3:726216d4:::smithi10536496-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.517426401000002, 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.10104360799999999, 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 166 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185519+0000", 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185520+0000", 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4099999999999999e-07 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185522+0000", 2024-09-15T03:42:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.588e-06 2024-09-15T03:42:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185526+0000", 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5130000000000001e-06 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185528+0000", 2024-09-15T03:42:05.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3099999999999999e-06 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186513+0000", 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098462300000000001 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186571+0000", 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8233000000000003e-05 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.186810+0000", 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023878899999999999 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.187938+0000", 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001128698 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286460+0000", 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098521759 2024-09-15T03:42:05.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286563+0000", 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102654 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:167 3.es0 3:726216d4:::smithi10536496-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.517407128999999, 2024-09-15T03:42:05.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.101989948, 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 167 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185538+0000", 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185539+0000", 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8599999999999999e-07 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185540+0000", 2024-09-15T03:42:05.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4020000000000001e-06 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185544+0000", 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6619999999999998e-06 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.185547+0000", 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6330000000000002e-06 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188004+0000", 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024565619999999998 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.188048+0000", 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3925999999999999e-05 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286831+0000", 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.098782707999999997 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.286925+0000", 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4609000000000002e-05 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.287528+0000", 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00060316000000000005 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.754 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.10s1 45/44 ECSubWrite(tid=39, reqid=client.4562.0:287, at_version=45'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 53.645815777999999, 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30713189699999999, 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057130+0000", 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057131+0000", 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3799999999999996e-07 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057345+0000", 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021436799999999999 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362921+0000", 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.30557607399999998 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362927+0000", 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8549999999999998e-06 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362967+0000", 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9654999999999999e-05 2024-09-15T03:42:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.362974+0000", 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5560000000000002e-06 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.364234+0000", 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012599200000000001 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.364262+0000", 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7631e-05 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.es0 54/32 ECSubWriteReply(tid=55, last_complete=54'26, committed=1, applied=1))", 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 44.645275449000003, 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.195800472, 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057670+0000", 2024-09-15T03:42:05.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.057673+0000", 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.751e-06 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253340+0000", 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.19566710400000001 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253347+0000", 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1250000000000004e-06 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253426+0000", 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8955999999999994e-05 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:21.253471+0000", 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4121e-05 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.763 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:546 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~0,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112325+0000", 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.590620627, 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90424344599999995, 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 546 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112325+0000", 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112325+0000", 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112327+0000", 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.227e-06 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112339+0000", 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1459e-05 2024-09-15T03:42:05.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112363+0000", 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4369999999999999e-05 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112370+0000", 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0509999999999999e-06 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112424+0000", 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3996000000000003e-05 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112427+0000", 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1e-06 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016449+0000", 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90402216700000004 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016568+0000", 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000119076 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:547 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.590563794000001, 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90430617099999999, 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 547 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:42:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112382+0000", 2024-09-15T03:42:05.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6700000000000003e-07 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112386+0000", 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7840000000000001e-06 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112396+0000", 2024-09-15T03:42:05.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0111e-05 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112399+0000", 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0199999999999999e-06 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112459+0000", 2024-09-15T03:42:05.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9280999999999997e-05 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112461+0000", 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9530000000000002e-06 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016597+0000", 2024-09-15T03:42:05.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90413675000000004 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016688+0000", 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0605000000000004e-05 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.777 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:548 3.es0 3:71615b7a:::smithi10536496-50:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.590528900999999, 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.90445608300000002, 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 548 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112417+0000", 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4000000000000002e-07 2024-09-15T03:42:05.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112420+0000", 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7499999999999999e-06 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112427+0000", 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8809999999999998e-06 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112430+0000", 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5999999999999998e-06 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112476+0000", 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5525000000000002e-05 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112477+0000", 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016731+0000", 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.904254585 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:39.016873+0000", 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014156499999999999 2024-09-15T03:42:05.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s2 76/44 ECSubWrite(tid=93, reqid=client.4562.0:622, at_version=0'0, trim_to=0'0, roll_forward_to=76'43))", 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:43.025659+0000", 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.677286135999999, 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.53751148000000004, 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.025659+0000", 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.025659+0000", 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.025660+0000", 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3099999999999996e-07 2024-09-15T03:42:05.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.025672+0000", 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1562999999999999e-05 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.562364+0000", 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.53669215100000001 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.562376+0000", 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2286999999999999e-05 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.562525+0000", 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014837800000000001 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.562554+0000", 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000999999999999e-05 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.563136+0000", 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058246000000000003 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:43.563171+0000", 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4808999999999998e-05 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:629 3.es0 3:7dd49206:::smithi10536496-20:head [copy-from ver 0] snapc 28=[28,27,26,25] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.022015+0000", 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.680930365999998, 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01355375, 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 629 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022015+0000", 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022015+0000", 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022016+0000", 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.187e-06 2024-09-15T03:42:05.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022022+0000", 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2669999999999997e-06 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022032+0000", 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0003000000000001e-05 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022037+0000", 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.152e-06 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022090+0000", 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3208999999999998e-05 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022098+0000", 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1089999999999998e-06 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.025216+0000", 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031174610000000002 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.025349+0000", 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013370100000000001 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.035569+0000", 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010219661 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:630 3.es0 3:7dd49206:::smithi10536496-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.022046+0000", 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.680899587999999, 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014069823, 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 630 2024-09-15T03:42:05.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022046+0000", 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022046+0000", 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022046+0000", 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6800000000000002e-07 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022048+0000", 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6080000000000001e-06 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022053+0000", 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0749999999999997e-06 2024-09-15T03:42:05.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022056+0000", 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6599999999999999e-06 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022145+0000", 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9944999999999993e-05 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.022151+0000", 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2719999999999997e-06 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.025471+0000", 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003320206 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for blocked object", 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.025503+0000", 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2143e-05 2024-09-15T03:42:05.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.035953+0000", 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010450265 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.036018+0000", 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4892000000000001e-05 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.036116+0000", 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7489000000000001e-05 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs1 77/32 ECSubWrite(tid=144, reqid=osd.11.0:144, at_version=77'38, trim_to=0'0, roll_forward_to=75'36))", 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.031578+0000", 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.671367610000001, 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012567929, 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.031578+0000", 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.031578+0000", 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.031579+0000", 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4099999999999997e-07 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.031602+0000", 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3056000000000001e-05 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032589+0000", 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00098730600000000008 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032809+0000", 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220275 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032870+0000", 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0220000000000003e-05 2024-09-15T03:42:05.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032879+0000", 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6039999999999999e-06 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.044121+0000", 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011242078000000001 2024-09-15T03:42:05.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.044146+0000", 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4549000000000002e-05 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs1 77/32 ECSubWrite(tid=145, reqid=osd.11.0:145, at_version=77'40, trim_to=0'0, roll_forward_to=75'36))", 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.032543+0000", 2024-09-15T03:42:05.804 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.670403013000001, 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017191570999999999, 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032543+0000", 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032543+0000", 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032543+0000", 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1399999999999997e-07 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032574+0000", 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0774e-05 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033481+0000", 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00090670799999999999 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.033489+0000", 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0569999999999993e-06 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.035191+0000", 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001702255 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:05.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.035209+0000", 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7909000000000001e-05 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.049693+0000", 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014484212 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.049734+0000", 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1041999999999998e-05 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.es0 77/71 ECSubWriteReply(tid=89, last_complete=77'59, committed=1, applied=1))", 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.039186+0000", 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.663759569, 2024-09-15T03:42:05.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011780687, 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.039186+0000", 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.039186+0000", 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.039187+0000", 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3099999999999996e-07 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.039193+0000", 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8200000000000002e-06 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.050360+0000", 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011167587 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.050367+0000", 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7449999999999998e-06 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.050424+0000", 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7262000000000002e-05 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.050967+0000", 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054244199999999997 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.es0 77/71 ECSubWriteReply(tid=90, last_complete=77'59, committed=1, applied=1))", 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.053028+0000", 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.649917998999999, 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.60035057000000003, 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.053028+0000", 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.053028+0000", 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.053028+0000", 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7700000000000004e-07 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.053033+0000", 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.853e-06 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.653184+0000", 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.60015081100000001 2024-09-15T03:42:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.653195+0000", 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0933e-05 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.653323+0000", 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000127708 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.653378+0000", 2024-09-15T03:42:05.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5587999999999997e-05 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.es0 78/71 ECSubWriteReply(tid=93, last_complete=78'63, committed=1, applied=1))", 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:45.047670+0000", 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.655275482, 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.278538022, 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.047670+0000", 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.047670+0000", 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:05.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.047671+0000", 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8100000000000002e-07 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.047675+0000", 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4649999999999996e-06 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:05.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.326093+0000", 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.278417839 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.326099+0000", 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9410000000000004e-06 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.326183+0000", 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4097999999999993e-05 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:05.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:45.326208+0000", 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4898000000000001e-05 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:05.823 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:05.824 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing write tid 5 to smithi10536496-26 2024-09-15T03:42:05.825 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing write tid 6 to smithi10536496-26 2024-09-15T03:42:05.825 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 82 (ObjNum 132 snap 32 seq_num 132) dirty exists 2024-09-15T03:42:05.825 INFO:tasks.rados.rados.0.smithi105.stdout:343: left oid 26 (ObjNum 132 snap 32 seq_num 132) 2024-09-15T03:42:05.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:42:05.826 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:05.824+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:05.924 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:05.924 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:42:05.924 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:05.925 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:42:05.924+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:42:05.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:42:06.026 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:06.024+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:06.058 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-09-15T03:42:06.059 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:42:06.059 INFO:tasks.rados.rados.0.smithi105.stdout:343: done (5 left) 2024-09-15T03:42:06.059 INFO:tasks.rados.rados.0.smithi105.stdout:344: done (4 left) 2024-09-15T03:42:06.059 INFO:tasks.rados.rados.0.smithi105.stdout:347: done (3 left) 2024-09-15T03:42:06.060 INFO:tasks.rados.rados.0.smithi105.stdout:348: done (2 left) 2024-09-15T03:42:06.060 INFO:tasks.rados.rados.0.smithi105.stdout:349: done (1 left) 2024-09-15T03:42:06.060 INFO:tasks.rados.rados.0.smithi105.stdout:350: done (0 left) 2024-09-15T03:42:06.060 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 14 current snap is 33 2024-09-15T03:42:06.060 INFO:tasks.rados.rados.0.smithi105.stdout:351: seq_num 137 ranges {2703360=753664,3457024=786432,4243456=655360,4898816=114688} 2024-09-15T03:42:06.072 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.072 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:42:06.073 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:42:06.073 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-15T03:42:06.077 INFO:tasks.rados.rados.0.smithi105.stdout:351: writing smithi10536496-14 from 2703360 to 3457024 tid 1 2024-09-15T03:42:06.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:42:06.082 INFO:tasks.rados.rados.0.smithi105.stdout:351: writing smithi10536496-14 from 3457024 to 4243456 tid 2 2024-09-15T03:42:06.085 INFO:tasks.rados.rados.0.smithi105.stdout:351: writing smithi10536496-14 from 4243456 to 4898816 tid 3 2024-09-15T03:42:06.086 INFO:tasks.rados.rados.0.smithi105.stdout:351: writing smithi10536496-14 from 4898816 to 5013504 tid 4 2024-09-15T03:42:06.086 INFO:tasks.rados.rados.0.smithi105.stdout:352: snap_remove snap 30 2024-09-15T03:42:06.161 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:06.160+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:06.181 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:42:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:42:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:42:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.181 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 34/32 ECSubWrite(tid=0, reqid=client.4562.0:13, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.159829848000001, 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011510825000000001, 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:42:06.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019688+0000", 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019689+0000", 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5799999999999998e-07 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019738+0000", 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8652000000000001e-05 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019844+0000", 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106623 2024-09-15T03:42:06.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019852+0000", 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019952+0000", 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100111 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019961+0000", 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3689999999999996e-06 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031174+0000", 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011212744 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031199+0000", 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5278000000000001e-05 2024-09-15T03:42:06.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.159768249999999, 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011454499999999999, 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019750+0000", 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.019805+0000", 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5062000000000002e-05 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020203+0000", 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039759399999999997 2024-09-15T03:42:06.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020209+0000", 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6189999999999999e-06 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020263+0000", 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4410000000000003e-05 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.020283+0000", 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0176000000000001e-05 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031185+0000", 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010901569 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031204+0000", 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9663000000000001e-05 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.157920391000005, 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012719667, 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021598+0000", 2024-09-15T03:42:06.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021599+0000", 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0500000000000003e-07 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021655+0000", 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6644999999999997e-05 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021703+0000", 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7549999999999997e-05 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021711+0000", 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1519999999999992e-06 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022571+0000", 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086020699999999999 2024-09-15T03:42:06.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022577+0000", 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3339999999999998e-06 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034283+0000", 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011705224 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034317+0000", 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.485e-05 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.157197960999994, 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012060876, 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022320+0000", 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.022321+0000", 2024-09-15T03:42:06.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4200000000000005e-07 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023195+0000", 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087440700000000001 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023229+0000", 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3905000000000001e-05 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023237+0000", 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.43e-06 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023278+0000", 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1137999999999998e-05 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.023286+0000", 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5379999999999997e-06 2024-09-15T03:42:06.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034356+0000", 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01107022 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.034381+0000", 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5395999999999999e-05 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=5, reqid=client.4562.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.153185182000001, 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018217534000000001, 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026333+0000", 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026334+0000", 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8500000000000001e-07 2024-09-15T03:42:06.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.026371+0000", 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6835999999999998e-05 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036697+0000", 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010326484 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036758+0000", 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0989999999999997e-05 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036813+0000", 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5050000000000003e-05 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.036822+0000", 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3149999999999998e-06 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044526+0000", 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077039439999999999 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.044551+0000", 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.423e-05 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.206 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/32 ECSubWrite(tid=10, reqid=client.4562.0:53, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.142353654999994, 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087714329999999993, 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037165+0000", 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5799999999999999e-07 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037181+0000", 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5954999999999999e-05 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122876+0000", 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085695108000000006 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122883+0000", 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4069999999999998e-06 2024-09-15T03:42:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122927+0000", 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552999999999998e-05 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.122937+0000", 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.006e-05 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124855+0000", 2024-09-15T03:42:06.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001917687 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.124879+0000", 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4002000000000002e-05 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:305 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:06.211 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.161653483999999, 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057908950000001, 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 305 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017865+0000", 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017866+0000", 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.468e-06 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017871+0000", 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8949999999999997e-06 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017882+0000", 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0781000000000001e-05 2024-09-15T03:42:06.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017888+0000", 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7030000000000003e-06 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018332+0000", 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000444633 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018335+0000", 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7690000000000001e-06 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023618+0000", 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.005282727 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023656+0000", 2024-09-15T03:42:06.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7919000000000001e-05 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:306 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.161621474, 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0057866579999999, 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 306 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017897+0000", 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7800000000000002e-07 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017900+0000", 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.756e-06 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017907+0000", 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6510000000000004e-06 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017912+0000", 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4009999999999996e-06 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018381+0000", 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000468696 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018383+0000", 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.906e-06 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023676+0000", 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0052937150000001 2024-09-15T03:42:06.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023683+0000", 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0550000000000002e-06 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:307 3.10s1 3.e99e4070 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.161601853999997, 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.00579789, 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 307 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.223 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017916+0000", 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017917+0000", 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7399999999999999e-07 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017920+0000", 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3500000000000001e-06 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017926+0000", 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8529999999999997e-06 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.017929+0000", 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3100000000000001e-06 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018419+0000", 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00048992199999999995 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:14.018420+0000", 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.313e-06 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023709+0000", 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.005288537 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:15.023714+0000", 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2310000000000004e-06 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s6 50/44 ECSubWrite(tid=35, reqid=client.4562.0:325, at_version=50'24, trim_to=0'0, roll_forward_to=50'20))", 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 49.093358379999998, 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075270610000000002, 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:42:06.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086160+0000", 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.086179+0000", 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8609000000000001e-05 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160878+0000", 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074699163999999998 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160884+0000", 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6760000000000004e-06 2024-09-15T03:42:06.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160935+0000", 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.117e-05 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.160948+0000", 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3355e-05 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.161407+0000", 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045902099999999999 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.161430+0000", 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2968000000000001e-05 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDScrubReserve(3.7s1 REQUEST e60)", 2024-09-15T03:42:06.232 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 38.710494316999998, 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120973636, 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469024+0000", 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3400000000000001e-07 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.469026+0000", 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3530000000000002e-06 2024-09-15T03:42:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589898+0000", 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120871944 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589906+0000", 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5660000000000001e-06 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589955+0000", 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9027999999999999e-05 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589973+0000", 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8318e-05 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:27.589998+0000", 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4292999999999999e-05 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.236 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=62, reqid=client.4562.0:438, at_version=62'25, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.138030815, 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017875489000000001, 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041487+0000", 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.041488+0000", 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.03e-07 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042539+0000", 2024-09-15T03:42:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001050823 2024-09-15T03:42:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042580+0000", 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0908000000000002e-05 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042587+0000", 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7929999999999998e-06 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042636+0000", 2024-09-15T03:42:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8996999999999999e-05 2024-09-15T03:42:06.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.042645+0000", 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0899999999999994e-06 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059315+0000", 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016669970999999999 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059363+0000", 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7503999999999998e-05 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=63, reqid=client.4562.0:439, at_version=62'26, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.133819367000001, 2024-09-15T03:42:06.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013689702, 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:42:06.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045699+0000", 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045881+0000", 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018117900000000001 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045908+0000", 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7741000000000001e-05 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.045913+0000", 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0370000000000002e-06 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047115+0000", 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012019629999999999 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047126+0000", 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0210000000000001e-05 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059370+0000", 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01224459 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059389+0000", 2024-09-15T03:42:06.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8328000000000001e-05 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=64, reqid=client.4562.0:440, at_version=62'27, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.133216666000003, 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016955748, 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:42:06.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046302+0000", 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046339+0000", 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7045999999999997e-05 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046361+0000", 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2115000000000001e-05 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.046366+0000", 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2290000000000002e-06 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047844+0000", 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001477864 2024-09-15T03:42:06.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047853+0000", 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6559999999999998e-06 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063237+0000", 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015383681 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063257+0000", 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0645999999999999e-05 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.bs4 62/32 ECSubWrite(tid=96, reqid=client.4562.0:446, at_version=0'0, trim_to=0'0, roll_forward_to=62'18))", 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.132014724999998, 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011862770999999999, 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047503+0000", 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047504+0000", 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5600000000000004e-07 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047516+0000", 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1987999999999999e-05 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047611+0000", 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4741000000000007e-05 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047622+0000", 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1270999999999999e-05 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047672+0000", 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.99e-05 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047718+0000", 2024-09-15T03:42:06.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5705999999999999e-05 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059338+0000", 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011619643000000001 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.059366+0000", 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8666e-05 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 62/44 ECSubWrite(tid=65, reqid=client.4562.0:441, at_version=62'28, trim_to=0'0, roll_forward_to=58'23))", 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.131637943999998, 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015392342999999999, 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047880+0000", 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047881+0000", 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8700000000000004e-07 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047900+0000", 2024-09-15T03:42:06.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8895000000000001e-05 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047932+0000", 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1656999999999999e-05 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.047939+0000", 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8329999999999996e-06 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051450+0000", 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035115120000000001 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051461+0000", 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1439e-05 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063261+0000", 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011799924999999999 2024-09-15T03:42:06.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063273+0000", 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1195e-05 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.0s5 62/44 ECSubWrite(tid=41, reqid=client.4562.0:443, at_version=62'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.130233820000001, 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014072049999999999, 2024-09-15T03:42:06.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049284+0000", 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049285+0000", 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6300000000000001e-07 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049336+0000", 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1023000000000003e-05 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049377+0000", 2024-09-15T03:42:06.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1591000000000002e-05 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.049382+0000", 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9609999999999996e-06 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051693+0000", 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023109720000000001 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.051702+0000", 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9430000000000006e-06 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063337+0000", 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011634698000000001 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:29.063356+0000", 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9599e-05 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.266 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s6 95/82 ECSubWrite(tid=23, reqid=client.4562.0:794, at_version=95'56, trim_to=0'0, roll_forward_to=94'51))", 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:02.037567+0000", 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1419510639999997, 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.26915473899999998, 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.037567+0000", 2024-09-15T03:42:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.037567+0000", 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.037568+0000", 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0800000000000004e-07 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.037584+0000", 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6263e-05 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.305984+0000", 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.26839977399999998 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.305997+0000", 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.274e-05 2024-09-15T03:42:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.306083+0000", 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6649000000000003e-05 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.306102+0000", 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8652999999999999e-05 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.306697+0000", 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00059459300000000003 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:02.306722+0000", 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5259e-05 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 98/44 ECSubWrite(tid=184, reqid=client.4562.0:836, at_version=98'83, trim_to=0'0, roll_forward_to=98'80))", 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:05.058819+0000", 2024-09-15T03:42:06.271 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.120698865, 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011514893999999999, 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.058819+0000", 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.058819+0000", 2024-09-15T03:42:06.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.058820+0000", 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2799999999999995e-07 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059054+0000", 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023349699999999999 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059889+0000", 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00083564399999999997 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059896+0000", 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.737e-06 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.062029+0000", 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021326130000000002 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.062042+0000", 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3175e-05 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.070281+0000", 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082387810000000006 2024-09-15T03:42:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.070334+0000", 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3619000000000003e-05 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.8s6 98/44 ECSubWrite(tid=185, reqid=client.4562.0:837, at_version=98'84, trim_to=0'0, roll_forward_to=98'80))", 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:05.059838+0000", 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.1196806020000001, 2024-09-15T03:42:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.60945116799999999, 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059838+0000", 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059838+0000", 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059838+0000", 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4399999999999999e-07 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.059869+0000", 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0536e-05 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.667930+0000", 2024-09-15T03:42:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.60806089299999999 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.667942+0000", 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2191000000000001e-05 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.667985+0000", 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3189000000000003e-05 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.667994+0000", 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2609999999999993e-06 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.669260+0000", 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001266484 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.669289+0000", 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8770000000000001e-05 2024-09-15T03:42:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.281 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.282 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:42:06.283 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:06.281+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:06.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:06.381 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:42:06.381 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.383 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:06.383+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:06.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:42:06.484 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:06.483+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:06.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.486 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:06.486 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:42:06.486 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:42:06.486 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:42:06.585 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:06.585+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:300 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~442368] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:42:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.582164634000002, 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0389183930000001, 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 300 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:42:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009243+0000", 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009245+0000", 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.556e-06 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009830+0000", 2024-09-15T03:42:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058575800000000003 2024-09-15T03:42:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009844+0000", 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3247e-05 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009851+0000", 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4850000000000003e-06 2024-09-15T03:42:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010438+0000", 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058667099999999998 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010441+0000", 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2119999999999999e-06 2024-09-15T03:42:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031346+0000", 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0209051899999999 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.031477+0000", 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000130712 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.032397+0000", 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091970699999999997 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037882+0000", 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054851639999999998 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048077+0000", 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010194896 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.048161+0000", 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4795000000000004e-05 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:301 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~720896] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.581540050999998, 2024-09-15T03:42:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0456429519999999, 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 301 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.009868+0000", 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010594+0000", 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072566600000000003 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010605+0000", 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1228e-05 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010611+0000", 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8390000000000003e-06 2024-09-15T03:42:06.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010679+0000", 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7966999999999995e-05 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010681+0000", 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1409999999999999e-06 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033259+0000", 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.022577372 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.033385+0000", 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012635200000000001 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.034673+0000", 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001287853 2024-09-15T03:42:06.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038109+0000", 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003436508 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055462+0000", 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017353200999999999 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055511+0000", 2024-09-15T03:42:06.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8217999999999998e-05 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:302 3.11s0 3:8f673fa9:::smithi10536496-24:head [append 0~49152] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.580789670000001, 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.045100688, 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.608 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 302 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010618+0000", 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010619+0000", 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0699999999999996e-07 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010663+0000", 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4552000000000003e-05 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010680+0000", 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6175999999999999e-05 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010685+0000", 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8619999999999999e-06 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010726+0000", 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0259000000000002e-05 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010727+0000", 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.753e-06 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035776+0000", 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0250488879999999 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.035909+0000", 2024-09-15T03:42:06.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013254900000000001 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.036628+0000", 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00071956899999999998 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038283+0000", 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016543490000000001 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055650+0000", 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017367301000000002 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055719+0000", 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8622999999999997e-05 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:303 3.11s0 3:8f673fa9:::smithi10536496-24:head [setxattr _header (56) in=7b] snapc c=[c,b] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.580711784000002, 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0452317200000001, 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 303 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010696+0000", 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010697+0000", 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9400000000000005e-07 2024-09-15T03:42:06.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010700+0000", 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2509999999999999e-06 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010706+0000", 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4989999999999999e-06 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010711+0000", 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2250000000000002e-06 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010765+0000", 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4784000000000003e-05 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010767+0000", 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4160000000000001e-06 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037038+0000", 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.026271521 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037181+0000", 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014324800000000001 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037675+0000", 2024-09-15T03:42:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049306399999999998 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.039474+0000", 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017996729999999999 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055837+0000", 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016362393999999999 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.055928+0000", 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1051000000000006e-05 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:304 3.11s0 3:8f673fa9:::smithi10536496-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.580692861999999, 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.050398215, 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 304 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:42:06.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010715+0000", 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010717+0000", 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.125e-06 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010724+0000", 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6880000000000002e-06 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010728+0000", 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.777e-06 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010795+0000", 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7277e-05 2024-09-15T03:42:06.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:16.010796+0000", 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3230000000000001e-06 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.037918+0000", 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.02712114 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:06.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.038025+0000", 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010774 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056315+0000", 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.018289716000000001 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.056417+0000", 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102336 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061113+0000", 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046956749999999998 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:317 3.11s0 3:8b799747:::smithi10536496-30:head [rollback e] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 49.532327387999999, 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010406076, 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 317 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059080+0000", 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059081+0000", 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4099999999999998e-07 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059084+0000", 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.151e-06 2024-09-15T03:42:06.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059092+0000", 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4989999999999997e-06 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059096+0000", 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4499999999999997e-06 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060096+0000", 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010001019999999999 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060189+0000", 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2320999999999999e-05 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.060639+0000", 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044999599999999998 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062024+0000", 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013859650000000001 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069431+0000", 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074061190000000001 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069486+0000", 2024-09-15T03:42:06.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5831999999999997e-05 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:318 3.11s0 3:8b799747:::smithi10536496-30:head [append 0~0] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 49.532306351999999, 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010504207999999999, 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 318 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059101+0000", 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059102+0000", 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6600000000000002e-07 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059104+0000", 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.215e-06 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059110+0000", 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0279999999999997e-06 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.059114+0000", 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3340000000000003e-06 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061223+0000", 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002108249 2024-09-15T03:42:06.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061279+0000", 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6418000000000001e-05 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.061697+0000", 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000418109 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.062208+0000", 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051137000000000003 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069565+0000", 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073569799999999999 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:17.069606+0000", 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0139000000000003e-05 2024-09-15T03:42:06.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:542 3.11s0 3:8f673fa9:::smithi10536496-24:head [setxattr _header (57) in=7b] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.479095926999999, 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83759245800000004, 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 542 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112312+0000", 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7700000000000004e-07 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112315+0000", 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7999999999999999e-06 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112323+0000", 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5630000000000003e-06 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112326+0000", 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.114769+0000", 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024421389999999999 2024-09-15T03:42:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.114833+0000", 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4281000000000003e-05 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.115220+0000", 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00038725499999999998 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.116192+0000", 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097158599999999998 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.949795+0000", 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83360352100000001 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.949904+0000", 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109096 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.644 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:543 3.11s0 3:8f673fa9:::smithi10536496-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.112335+0000", 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.479072560999999, 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83899146099999999, 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 543 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112335+0000", 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112335+0000", 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112336+0000", 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9900000000000001e-07 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112337+0000", 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.728e-06 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112344+0000", 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2609999999999999e-06 2024-09-15T03:42:06.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.112347+0000", 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7390000000000001e-06 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.116073+0000", 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037256780000000001 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:06.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.116139+0000", 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6052999999999999e-05 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.950344+0000", 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.83420500900000005 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.950463+0000", 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118498 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.951327+0000", 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00086409599999999996 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 77/32 ECSubWrite(tid=145, reqid=osd.11.0:145, at_version=77'40, trim_to=0'0, roll_forward_to=75'36))", 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:44.032545+0000", 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.558862887, 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010871377, 2024-09-15T03:42:06.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032545+0000", 2024-09-15T03:42:06.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032545+0000", 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032546+0000", 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032574+0000", 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8705999999999999e-05 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032606+0000", 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2181000000000002e-05 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032614+0000", 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9740000000000002e-06 2024-09-15T03:42:06.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032942+0000", 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032719200000000002 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.032952+0000", 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0693000000000001e-05 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.043373+0000", 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010420469 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:44.043416+0000", 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3414000000000001e-05 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 84/32 ECSubWrite(tid=174, reqid=client.4562.0:723, at_version=84'55, trim_to=0'0, roll_forward_to=81'52))", 2024-09-15T03:42:06.655 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.564850973, 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010574913, 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.026557+0000", 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-07 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027143+0000", 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058585600000000005 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027185+0000", 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2546000000000002e-05 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027194+0000", 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3690000000000007e-06 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027258+0000", 2024-09-15T03:42:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3995e-05 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.027269+0000", 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1503e-05 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.037103+0000", 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098336789999999997 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.037132+0000", 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8844999999999999e-05 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.13s6 94/82 ECSubWrite(tid=204, reqid=osd.11.0:204, at_version=94'64, trim_to=0'0, roll_forward_to=91'62))", 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.022355+0000", 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5690530430000003, 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014064093, 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.022355+0000", 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.022355+0000", 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.022355+0000", 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8799999999999999e-07 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.022372+0000", 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7112000000000002e-05 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.022584+0000", 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000212322 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.022590+0000", 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6629999999999998e-06 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023014+0000", 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042389499999999998 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023024+0000", 2024-09-15T03:42:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7459999999999995e-06 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036395+0000", 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013371437999999999 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036419+0000", 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3728999999999999e-05 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.es1 94/71 ECSubWrite(tid=117, reqid=osd.2.0:117, at_version=94'83, trim_to=0'0, roll_forward_to=93'79))", 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.023648+0000", 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5677595809999998, 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012758064, 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023648+0000", 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023648+0000", 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023651+0000", 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6350000000000002e-06 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023667+0000", 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6419999999999999e-05 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023703+0000", 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5459000000000001e-05 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023709+0000", 2024-09-15T03:42:06.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4350000000000002e-06 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023743+0000", 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3414000000000002e-05 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023750+0000", 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6699999999999994e-06 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036387+0000", 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012636507999999999 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036406+0000", 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9522999999999999e-05 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.669 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.17s2 94/51 ECSubWrite(tid=166, reqid=osd.5.0:166, at_version=94'64, trim_to=0'0, roll_forward_to=91'62))", 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.023730+0000", 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5676776969999997, 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012617666, 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023730+0000", 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023730+0000", 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023730+0000", 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.66e-07 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023749+0000", 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8505e-05 2024-09-15T03:42:06.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023776+0000", 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7614e-05 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023781+0000", 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.135e-06 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023819+0000", 2024-09-15T03:42:06.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7228000000000003e-05 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.023829+0000", 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0555e-05 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036320+0000", 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012490894000000001 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036348+0000", 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7569e-05 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 94/32 ECSubWrite(tid=205, reqid=osd.11.0:205, at_version=94'79, trim_to=0'0, roll_forward_to=91'77))", 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.024399+0000", 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5670082570000003, 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01196854, 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024399+0000", 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024399+0000", 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024400+0000", 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3099999999999997e-07 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024422+0000", 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2209e-05 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024454+0000", 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1602e-05 2024-09-15T03:42:06.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024461+0000", 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6680000000000004e-06 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024501+0000", 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0711999999999999e-05 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.024511+0000", 2024-09-15T03:42:06.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4329999999999993e-06 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036353+0000", 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011842353999999999 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.036368+0000", 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4831000000000001e-05 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.17s2 94/51 ECSubWrite(tid=167, reqid=osd.5.0:167, at_version=94'66, trim_to=0'0, roll_forward_to=91'62))", 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.025500+0000", 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5659078649999998, 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017047667999999998, 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025500+0000", 2024-09-15T03:42:06.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025500+0000", 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025501+0000", 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1600000000000001e-07 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025521+0000", 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0893000000000001e-05 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025553+0000", 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1338999999999999e-05 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025559+0000", 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2110000000000003e-06 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025597+0000", 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8002e-05 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.025608+0000", 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0507e-05 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.042534+0000", 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016926165 2024-09-15T03:42:06.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.042548+0000", 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3835000000000001e-05 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.fs5 94/32 ECSubWrite(tid=206, reqid=osd.11.0:206, at_version=94'81, trim_to=0'0, roll_forward_to=91'77))", 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.026322+0000", 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5650856199999996, 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016206952, 2024-09-15T03:42:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026322+0000", 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026322+0000", 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026323+0000", 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.89e-07 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026341+0000", 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8360999999999999e-05 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026366+0000", 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4808e-05 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026371+0000", 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4639999999999997e-06 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026412+0000", 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0608999999999997e-05 2024-09-15T03:42:06.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.026421+0000", 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5959999999999993e-06 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.042504+0000", 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.016082685999999999 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.042529+0000", 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4939e-05 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:848 3.11s0 3:8b799747:::smithi10536496-30:head [read 0~278528,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:05.178312+0000", 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.413095274, 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010414471999999999, 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:06.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 848 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178312+0000", 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178312+0000", 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178314+0000", 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8560000000000001e-06 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178326+0000", 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1375999999999999e-05 2024-09-15T03:42:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178345+0000", 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9491e-05 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178352+0000", 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3600000000000001e-06 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178414+0000", 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2607999999999994e-05 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178529+0000", 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011497699999999999 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.188727+0000", 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010197804 2024-09-15T03:42:06.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:849 3.11s0 3:8b799747:::smithi10536496-30:head [read 0~278528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:05.178364+0000", 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.4130436230000001, 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010808909, 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 849 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178364+0000", 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178364+0000", 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178365+0000", 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5899999999999996e-07 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178377+0000", 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2707999999999999e-05 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178386+0000", 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4689999999999997e-06 2024-09-15T03:42:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178390+0000", 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5839999999999996e-06 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.180894+0000", 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002503866 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.180996+0000", 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000101539 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.189173+0000", 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0081771840000000005 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:850 3.11s0 3:8b799747:::smithi10536496-30:head [read 0~278528,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:05.178417+0000", 2024-09-15T03:42:06.697 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.412991195, 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011030326999999999, 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 850 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178417+0000", 2024-09-15T03:42:06.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178417+0000", 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178417+0000", 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8699999999999995e-07 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178421+0000", 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5190000000000001e-06 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178427+0000", 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5760000000000002e-06 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.178431+0000", 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.569e-06 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.181877+0000", 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034465479999999998 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.181925+0000", 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7772000000000003e-05 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:05.189447+0000", 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075217560000000001 2024-09-15T03:42:06.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:06.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:42:06.704 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:06.702+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:06.804 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:06.803+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:06.842 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:06.843 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-09-15T03:42:06.843 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.851 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:42:06.904 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:06.904+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:06.947 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:06.947 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-09-15T03:42:06.947 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-09-15T03:42:06.947 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-09-15T03:42:06.947 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:06.955 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:42:07.006 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:07.005+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:36 3.0s0 3:0e027997:::smithi10536496-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.031151468999994, 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.046079387999999999, 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 36 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021983+0000", 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021984+0000", 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021986+0000", 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9850000000000001e-06 2024-09-15T03:42:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021992+0000", 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4409999999999998e-06 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.021996+0000", 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4120000000000001e-06 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025204+0000", 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032082600000000001 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.025267+0000", 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2945000000000001e-05 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037375+0000", 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012108373 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.037446+0000", 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0335999999999999e-05 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068063+0000", 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.030617210999999998 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:44 3.9s0 3:9992a557:::smithi10536496-15:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.024544222000003, 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.133216362, 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 44 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028591+0000", 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028595+0000", 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0429999999999999e-06 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028601+0000", 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2949999999999999e-06 2024-09-15T03:42:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028604+0000", 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6399999999999999e-06 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031289+0000", 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026843069999999999 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.031374+0000", 2024-09-15T03:42:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5649000000000006e-05 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032013+0000", 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00063854500000000002 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.039383+0000", 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0073696639999999997 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161755+0000", 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.122372592 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161807+0000", 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1875000000000001e-05 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:45 3.9s0 3:9992a557:::smithi10536496-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.024526608000002, 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.136258991, 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 45 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028608+0000", 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028609+0000", 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028611+0000", 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9e-06 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028616+0000", 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8690000000000001e-06 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.028620+0000", 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5159999999999999e-06 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.032997+0000", 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043766650000000001 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033054+0000", 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7055000000000002e-05 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161874+0000", 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12881989199999999 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161942+0000", 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8041999999999997e-05 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164867+0000", 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002925642 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=9, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.019801403000002, 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12849276700000001, 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033334+0000", 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000001e-07 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.033336+0000", 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7600000000000001e-06 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161667+0000", 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.12833182500000001 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161670+0000", 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7669999999999999e-06 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161709+0000", 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9303e-05 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161826+0000", 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116866 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:65 3.0s0 3:0215fa29:::smithi10536496-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.007219659, 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075129619999999994, 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 65 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045915+0000", 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045916+0000", 2024-09-15T03:42:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8500000000000002e-07 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045918+0000", 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4530000000000001e-06 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045922+0000", 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4790000000000001e-06 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045924+0000", 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.249e-06 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046887+0000", 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096257700000000005 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.046936+0000", 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9348000000000003e-05 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047414+0000", 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047808699999999998 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.050439+0000", 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003024555 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120968+0000", 2024-09-15T03:42:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070529644000000002 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121045+0000", 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6742999999999994e-05 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:66 3.0s0 3:0215fa29:::smithi10536496-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.007208117000005, 2024-09-15T03:42:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117527613, 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 66 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045927+0000", 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.35e-07 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045928+0000", 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045931+0000", 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2619999999999999e-06 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.045934+0000", 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4669999999999998e-06 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047770+0000", 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018362389999999999 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.047848+0000", 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7600000000000002e-05 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121082+0000", 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.073234457000000003 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121135+0000", 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.304e-05 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163454+0000", 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042319333000000001 2024-09-15T03:42:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:85 3.0s0 3:017cb3f5:::smithi10536496-29:head [append 0~475136,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.988016490000007, 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.060288026000000002, 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 85 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065118+0000", 2024-09-15T03:42:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065120+0000", 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2500000000000001e-06 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065572+0000", 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045274699999999998 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065585+0000", 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065591+0000", 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3539999999999996e-06 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065641+0000", 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9366000000000001e-05 2024-09-15T03:42:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065726+0000", 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5017000000000003e-05 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066267+0000", 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054127000000000005 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070477+0000", 2024-09-15T03:42:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042095300000000004 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125356+0000", 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.054879766000000003 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125406+0000", 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0114999999999997e-05 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:86 3.0s0 3:017cb3f5:::smithi10536496-29:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.987532595000005, 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.096114519999999995, 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 86 2024-09-15T03:42:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065602+0000", 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065603+0000", 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.99e-07 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065606+0000", 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.428e-06 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065618+0000", 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1474000000000001e-05 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065622+0000", 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6890000000000002e-06 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066858+0000", 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012361060000000001 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.066949+0000", 2024-09-15T03:42:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0906e-05 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067344+0000", 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039518300000000001 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.070507+0000", 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003163126 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161634+0000", 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.091127535999999995 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161717+0000", 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2472999999999995e-05 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:87 3.0s0 3:017cb3f5:::smithi10536496-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.987490952000002, 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.100018627, 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 87 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065644+0000", 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.15e-07 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065646+0000", 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065650+0000", 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0520000000000003e-06 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.065652+0000", 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3889999999999999e-06 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067593+0000", 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00194067 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067669+0000", 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5801000000000002e-05 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162263+0000", 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.094593951999999995 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162384+0000", 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012107700000000001 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165663+0000", 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032787480000000002 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=16, last_complete=35'8, committed=1, applied=1))", 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.985237505000001, 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.053160594999999998, 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067897+0000", 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067898+0000", 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.067902+0000", 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.658e-06 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120872+0000", 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.052970020999999999 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120876+0000", 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0409999999999999e-06 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.120905+0000", 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9064000000000002e-05 2024-09-15T03:42:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.121058+0000", 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153132 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.984249164000005, 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056528954999999999, 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068886+0000", 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6199999999999998e-07 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.068889+0000", 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8119999999999999e-06 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125281+0000", 2024-09-15T03:42:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.056392146999999997 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125285+0000", 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4970000000000001e-06 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125309+0000", 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4561000000000001e-05 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.125415+0000", 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105476 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:160 3.9s0 3:9a083222:::smithi10536496-8:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.891766715000003, 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.078417015000000007, 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 160 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161368+0000", 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161369+0000", 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9999999999999995e-07 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161767+0000", 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00039813099999999997 2024-09-15T03:42:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161777+0000", 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0121e-05 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.161783+0000", 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.271e-06 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162253+0000", 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00047015000000000002 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.162324+0000", 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1173000000000004e-05 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.163003+0000", 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067870599999999995 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.164585+0000", 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001581814 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239746+0000", 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.075161221 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239785+0000", 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9428000000000003e-05 2024-09-15T03:42:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=23, last_complete=35'6, committed=1, applied=1))", 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.887799051000002, 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074459365999999999, 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165336+0000", 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6699999999999999e-07 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.165339+0000", 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3829999999999999e-06 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239636+0000", 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.074297228000000007 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239640+0000", 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8680000000000001e-06 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239691+0000", 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1447999999999999e-05 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:02.239795+0000", 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103972 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4562.0:287 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 8] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.024855461999998, 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.820123202, 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4562", 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 287 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028279+0000", 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028281+0000", 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5200000000000001e-06 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028286+0000", 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2809999999999999e-06 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028300+0000", 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3668e-05 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.028307+0000", 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9539999999999998e-06 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029951+0000", 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001644419 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.029955+0000", 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6260000000000001e-06 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037676+0000", 2024-09-15T03:42:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077205920000000001 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.037910+0000", 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234212 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.057184+0000", 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.019273795999999999 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.059033+0000", 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00184902 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848316+0000", 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78928352999999996 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848403+0000", 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6583999999999996e-05 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 45/44 ECSubWriteReply(tid=39, last_complete=45'16, committed=1, applied=1))", 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.988184103000002, 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78386887299999997, 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064951+0000", 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.064956+0000", 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848087+0000", 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.78313173400000002 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848099+0000", 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1173e-05 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848179+0000", 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0921999999999995e-05 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:12.848820+0000", 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064024899999999996 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.ds2 71/51 ECSubWrite(tid=50, reqid=client.4562.0:535, at_version=71'28, trim_to=0'0, roll_forward_to=71'24))", 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:38.052671+0000", 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.000463683, 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.230093292, 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052671+0000", 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052671+0000", 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052672+0000", 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.052692+0000", 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9967e-05 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282058+0000", 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.22936631900000001 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282064+0000", 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6330000000000001e-06 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282143+0000", 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9660999999999995e-05 2024-09-15T03:42:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282152+0000", 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8580000000000006e-06 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282739+0000", 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058726799999999995 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:38.282765+0000", 2024-09-15T03:42:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5075e-05 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 83/82 ECSubWriteReply(tid=81, last_complete=83'36, committed=1, applied=1))", 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:50.039554+0000", 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.013580982000001, 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.41583952099999999, 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039554+0000", 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039554+0000", 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039555+0000", 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-09-15T03:42:07.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.039558+0000", 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4039999999999999e-06 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.454878+0000", 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.41532027900000001 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.454883+0000", 2024-09-15T03:42:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7149999999999997e-06 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.454949+0000", 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6211999999999996e-05 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.455393+0000", 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000444289 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 83/82 ECSubWriteReply(tid=83, last_complete=83'36, committed=1, applied=1))", 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:50.042724+0000", 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.010410843999999, 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.96124359599999998, 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042724+0000", 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042724+0000", 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042725+0000", 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2500000000000005e-07 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.042730+0000", 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0549999999999999e-06 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003879+0000", 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.96114944499999999 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003884+0000", 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8400000000000002e-06 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003941+0000", 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6838e-05 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:51.003968+0000", 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6693e-05 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWrite(3.5s1 83/82 ECSubWrite(tid=11, reqid=client.4562.0:713, at_version=83'45, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 16.822172191, 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.26109879699999999, 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.230963+0000", 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.231156+0000", 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019295299999999999 2024-09-15T03:42:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490877+0000", 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.25972105600000001 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490882+0000", 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5750000000000002e-06 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490951+0000", 2024-09-15T03:42:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.923e-05 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.490958+0000", 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8549999999999996e-06 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.492040+0000", 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010815460000000001 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:41:50.492062+0000", 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1837000000000001e-05 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 94/71 ECSubWriteReply(tid=106, last_complete=94'53, committed=1, applied=1))", 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-09-15T03:42:01.046529+0000", 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.006605982, 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33711816700000002, 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.046529+0000", 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.046529+0000", 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.046529+0000", 2024-09-15T03:42:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3799999999999998e-07 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.046532+0000", 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0290000000000001e-06 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.382875+0000", 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.33634271900000001 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.382879+0000", 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9179999999999996e-06 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.382953+0000", 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3688000000000003e-05 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-09-15T03:42:01.383647+0000", 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00069437500000000001 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T03:42:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T03:42:07.151 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T03:42:07.153 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:352: done (1 left) 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:353: delete oid 43 current snap is 33 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing write tid 1 to smithi10536496-14 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing write tid 2 to smithi10536496-14 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing write tid 3 to smithi10536496-14 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing write tid 4 to smithi10536496-14 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing write tid 5 to smithi10536496-14 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing write tid 6 to smithi10536496-14 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 76 (ObjNum 137 snap 33 seq_num 137) dirty exists 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: left oid 14 (ObjNum 137 snap 33 seq_num 137) 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:351: done (1 left) 2024-09-15T03:42:07.154 INFO:tasks.rados.rados.0.smithi105.stdout:353: done (0 left) 2024-09-15T03:42:07.155 INFO:tasks.rados.rados.0.smithi105.stdout:354: delete oid 41 current snap is 33 2024-09-15T03:42:07.155 INFO:tasks.rados.rados.0.smithi105.stdout:354: done (0 left) 2024-09-15T03:42:07.155 INFO:tasks.rados.rados.0.smithi105.stdout:355: snap_create 2024-09-15T03:42:07.155 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:42:07.155 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-15T03:42:07.156 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:07.152+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:07.237 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:42:07.254 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:07.254+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:07.262 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:07.262 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:07.262 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-09-15T03:42:07.262 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:07.271 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:42:07.355 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:07.354+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:07.369 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:07.370 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:07.370 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:42:07.370 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:42:07.370 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:07.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:42:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:42:07.381 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:42:07.393 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:42:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:42:07.413 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:42:07.413 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:42:07.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:42:07.432 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:42:07.451 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:42:07.452 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:42:07.456 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:07.455+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:07.472 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:42:07.472 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:42:07.480 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:07.480 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:42:07.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:42:07.480 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-09-15T03:42:07.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.480 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpRead(3.as1 35/32 ECSubRead(tid=8, to_read={3:5dde9339:::smithi10536496-16:head=0,4096,0}, subchunks={3:5dde9339:::smithi10536496-16:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:07.481 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:42:07.481 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.431797528999994, 2024-09-15T03:42:07.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010899076000000001, 2024-09-15T03:42:07.481 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.481 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046975+0000", 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046976+0000", 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0500000000000002e-07 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.046985+0000", 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6700000000000006e-06 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057489+0000", 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010503530000000001 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057501+0000", 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2311e-05 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057557+0000", 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5877000000000001e-05 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.057874+0000", 2024-09-15T03:42:07.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031688300000000001 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=10, reqid=client.4562.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.420168630000006, 2024-09-15T03:42:07.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024891555999999999, 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058604+0000", 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.47e-07 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.058624+0000", 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9358999999999999e-05 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082955+0000", 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024331663 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.082963+0000", 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3100000000000003e-06 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083025+0000", 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1793999999999998e-05 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083035+0000", 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0892e-05 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083478+0000", 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00044275899999999998 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.083495+0000", 2024-09-15T03:42:07.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6932e-05 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:93 3.ds0 3:baae61d4:::smithi10536496-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.402525130000001, 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036033748999999997, 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.490 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 93 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076247+0000", 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076248+0000", 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9200000000000001e-07 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076250+0000", 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076257+0000", 2024-09-15T03:42:07.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7089999999999996e-06 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.076261+0000", 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6360000000000002e-06 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078804+0000", 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025427230000000002 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.078880+0000", 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5731999999999999e-05 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084330+0000", 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0054505719999999999 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.084432+0000", 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101584 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112281+0000", 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027849032999999999 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/32 ECSubReadReply(tid=3, attrs_read=0))", 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.392699381, 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026222024, 2024-09-15T03:42:07.495 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086073+0000", 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3999999999999998e-07 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.086080+0000", 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3370000000000002e-06 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111977+0000", 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025896667000000002 2024-09-15T03:42:07.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.111987+0000", 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8349999999999999e-06 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112050+0000", 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3020000000000003e-05 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.112295+0000", 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000244925 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:126 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.363622348000007, 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011137536, 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:07.499 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 126 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:07.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115150+0000", 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115152+0000", 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0339999999999999e-06 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115156+0000", 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0219999999999998e-06 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.115159+0000", 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2229999999999999e-06 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117394+0000", 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002234977 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.117479+0000", 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4628000000000004e-05 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122860+0000", 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053811379999999997 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.122941+0000", 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0622000000000002e-05 2024-09-15T03:42:07.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126287+0000", 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033465779999999998 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.352705865000004, 2024-09-15T03:42:07.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017309390000000001, 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126066+0000", 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126067+0000", 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5400000000000001e-07 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.126072+0000", 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3920000000000002e-06 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143252+0000", 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017179104000000001 2024-09-15T03:42:07.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143257+0000", 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7470000000000002e-06 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143315+0000", 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7958000000000003e-05 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.143376+0000", 2024-09-15T03:42:07.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0535000000000002e-05 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=19, reqid=client.4562.0:144, at_version=35'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.332320711999998, 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015149222, 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146452+0000", 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.85e-07 2024-09-15T03:42:07.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.146470+0000", 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7309999999999999e-05 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160975+0000", 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014505123 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.160981+0000", 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6959999999999999e-06 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161035+0000", 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3517999999999998e-05 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161046+0000", 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.065e-05 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161571+0000", 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052504699999999995 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.161601+0000", 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0093000000000001e-05 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.as1 35/32 ECSubWrite(tid=23, reqid=client.4562.0:154, at_version=35'12, trim_to=0'0, roll_forward_to=35'11))", 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.313393314999999, 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.083287300999999994, 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165379+0000", 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165380+0000", 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0999999999999997e-07 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.165457+0000", 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6823999999999995e-05 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247921+0000", 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082464198000000002 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247926+0000", 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9880000000000004e-06 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247954+0000", 2024-09-15T03:42:07.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8306e-05 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.247960+0000", 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7570000000000001e-06 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248650+0000", 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000689816 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:02.248666+0000", 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6501999999999999e-05 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:308 3.13s1 3.1f172733 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.467908223999999, 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017692614, 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.517 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 308 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:42:07.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010864+0000", 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010866+0000", 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2680000000000001e-06 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010929+0000", 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3187000000000003e-05 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010944+0000", 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4701e-05 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.010952+0000", 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4610000000000002e-06 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011020+0000", 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8553000000000006e-05 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:14.011023+0000", 2024-09-15T03:42:07.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8150000000000002e-06 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028536+0000", 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.017512913 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:15.028557+0000", 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0716e-05 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:310 3.as1 3:5629793d:::smithi10536496-26:head [delete] snapc e=[e,c,b] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e49)", 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.473313105000003, 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0408854540000001, 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 310 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005459+0000", 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005461+0000", 2024-09-15T03:42:07.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.643e-06 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005465+0000", 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8589999999999998e-06 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005480+0000", 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5407999999999999e-05 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.005487+0000", 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4229999999999999e-06 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008383+0000", 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002895309 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:16.008386+0000", 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8949999999999998e-06 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.031743+0000", 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0233571130000001 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.032058+0000", 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000315123 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.033215+0000", 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011571579999999999 2024-09-15T03:42:07.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.037843+0000", 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046277380000000002 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.045957+0000", 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081145560000000002 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.046345+0000", 2024-09-15T03:42:07.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000387229 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 50/49 ECSubWrite(tid=151, reqid=client.4562.0:335, at_version=50'20, trim_to=0'0, roll_forward_to=50'18))", 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.348001046, 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067600538000000002, 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130771+0000", 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130773+0000", 2024-09-15T03:42:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2900000000000001e-06 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.130954+0000", 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018089000000000001 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196934+0000", 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065979530999999994 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.196943+0000", 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0499999999999997e-06 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197012+0000", 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8745999999999995e-05 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.197025+0000", 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2909e-05 2024-09-15T03:42:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198352+0000", 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001327188 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:17.198372+0000", 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9933999999999999e-05 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:532 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~442368] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.046777+0000", 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.431995129000001, 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011744466, 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 532 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.046777+0000", 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.046777+0000", 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.046779+0000", 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.562e-06 2024-09-15T03:42:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047019+0000", 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024031700000000001 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047032+0000", 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3385e-05 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047038+0000", 2024-09-15T03:42:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3149999999999999e-06 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047080+0000", 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2410000000000002e-05 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047153+0000", 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2538000000000002e-05 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.047668+0000", 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00051552100000000001 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.048400+0000", 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073139699999999995 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.058451+0000", 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010051164 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.058522+0000", 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0857000000000003e-05 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:533 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~524288] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.428736656000002, 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012117856, 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 533 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050036+0000", 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7599999999999996e-07 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050163+0000", 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012692000000000001 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050173+0000", 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8099999999999992e-06 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050177+0000", 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.676e-06 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050212+0000", 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4751e-05 2024-09-15T03:42:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050272+0000", 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9998999999999999e-05 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.050798+0000", 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052661299999999999 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052872+0000", 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0020738010000000001 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.062112+0000", 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0092395299999999993 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.062153+0000", 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1879999999999999e-05 2024-09-15T03:42:07.544 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:534 3.ds0 3:b9f64cc8:::smithi10536496-42:head [append 0~196608] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.051155+0000", 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.427616767, 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013798464999999999, 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.545 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 534 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051155+0000", 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051155+0000", 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051157+0000", 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1370000000000001e-06 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051316+0000", 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000159555 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051339+0000", 2024-09-15T03:42:07.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2555e-05 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051345+0000", 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.229e-06 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051453+0000", 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010767700000000001 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051515+0000", 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2150999999999995e-05 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051939+0000", 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042459399999999998 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.054404+0000", 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024641580000000001 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.064895+0000", 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010491097 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.064954+0000", 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9311999999999997e-05 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.550 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:535 3.ds0 3:b9f64cc8:::smithi10536496-42:head [setxattr _header (57) in=7b] snapc 22=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.427418693, 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.23179859899999999, 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 535 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051354+0000", 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051358+0000", 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3900000000000002e-06 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051364+0000", 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4710000000000004e-06 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051368+0000", 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7689999999999998e-06 2024-09-15T03:42:07.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052278+0000", 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00091056999999999998 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052376+0000", 2024-09-15T03:42:07.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7162000000000002e-05 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052640+0000", 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026434599999999999 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.057117+0000", 2024-09-15T03:42:07.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044771780000000001 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283090+0000", 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.22597297899999999 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283152+0000", 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2143000000000005e-05 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.556 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:536 3.ds0 3:b9f64cc8:::smithi10536496-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:38.051372+0000", 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.427400032000001, 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.232996072, 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 536 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.557 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051372+0000", 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051372+0000", 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051373+0000", 2024-09-15T03:42:07.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9999999999999998e-07 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051390+0000", 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7105999999999999e-05 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051399+0000", 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4709999999999996e-06 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.051403+0000", 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.155e-06 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052799+0000", 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013954620000000001 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.052867+0000", 2024-09-15T03:42:07.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8361000000000005e-05 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283472+0000", 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.23060489100000001 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.283578+0000", 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010643 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:38.284368+0000", 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00078979600000000005 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:634 3.ds0 3:b9f64cc8:::smithi10536496-42:28 [read 0~3555328,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:44.036221+0000", 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.442551022, 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011783227, 2024-09-15T03:42:07.562 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 634 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036221+0000", 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036221+0000", 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036222+0000", 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6199999999999997e-07 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.51e-06 2024-09-15T03:42:07.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036234+0000", 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5739999999999998e-06 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036239+0000", 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4599999999999996e-06 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036839+0000", 2024-09-15T03:42:07.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00060048700000000003 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036922+0000", 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2769000000000007e-05 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.048004+0000", 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011082665 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4562.0:635 3.ds0 3:b9f64cc8:::smithi10536496-42:28 [read 0~3555328,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.442528192000001, 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014992814, 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4562", 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 635 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.567 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036244+0000", 2024-09-15T03:42:07.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1600000000000002e-07 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036247+0000", 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6290000000000002e-06 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036252+0000", 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2279999999999998e-06 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.036256+0000", 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.303e-06 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.039208+0000", 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0029522020000000001 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.039315+0000", 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107315 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:44.051237+0000", 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011921721 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.10s2 78/49 ECSubWrite(tid=203, reqid=osd.13.0:203, at_version=78'56, trim_to=0'0, roll_forward_to=77'52))", 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:41:45.027703+0000", 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.451069200999999, 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.58712494400000004, 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.571 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027703+0000", 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027703+0000", 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027704+0000", 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4099999999999998e-07 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.027726+0000", 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1903999999999999e-05 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.613983+0000", 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.58625706499999997 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.613990+0000", 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7920000000000004e-06 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614056+0000", 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6383999999999993e-05 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614066+0000", 2024-09-15T03:42:07.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5889999999999992e-06 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614804+0000", 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073851799999999997 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:41:45.614828+0000", 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3951000000000002e-05 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDECSubOpWrite(3.17s4 99/51 ECSubWrite(tid=192, reqid=client.4562.0:856, at_version=99'76, trim_to=0'0, roll_forward_to=99'73))", 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-09-15T03:42:06.091694+0000", 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.387078499, 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.675184122, 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.091694+0000", 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.091694+0000", 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.091694+0000", 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1600000000000001e-07 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.091708+0000", 2024-09-15T03:42:07.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3582e-05 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.765957+0000", 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.67424903199999997 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.765981+0000", 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3652e-05 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-09-15T03:42:07.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.766032+0000", 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1020999999999998e-05 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_started", 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.766044+0000", 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2241e-05 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_committed", 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.766851+0000", 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080702400000000004 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-09-15T03:42:07.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-09-15T03:42:06.766878+0000", 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7053999999999999e-05 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-09-15T03:42:07.580 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:07.581 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:42:07.585 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:42:07.586 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:42:07.586 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:42:07.587 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:42:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:42:07.587 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:42:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:42:07.588 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:42:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:42:07.588 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:42:07.589 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:07.580+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:07.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:42:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:42:07.611 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:07.612 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:07.632 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:07.632 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:42:07.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:07.637+0000 7fea40e43640 0 -- 172.21.15.5:0/420566067 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7fea3c1951a0 msgr2=0x7fea3c197580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:07.658 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:07.658 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:42:07.677 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:07.678 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:07.682 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:07.681+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:07.724 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:07.724 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:07.733 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:42:07.782 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:07.782+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:07.831 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:07.831 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:07.839 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:42:07.883 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:42:07.883+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:42:07.978 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:07.978 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:07.985 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:07.984+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:07.988 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:42:08.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-15T03:42:08.079 INFO:tasks.rados.rados.0.smithi105.stdout:355: done (0 left) 2024-09-15T03:42:08.079 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 3 current snap is 34 2024-09-15T03:42:08.079 INFO:tasks.rados.rados.0.smithi105.stdout:356: seq_num 140 ranges {2850816=802816,3653632=425984,4079616=720896,4800512=409600} 2024-09-15T03:42:08.080 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.12 to 1 (10000) 2024-09-15T03:42:08.085 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:08.084+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:08.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-15T03:42:08.101 INFO:tasks.rados.rados.0.smithi105.stdout:356: writing smithi10536496-3 from 2850816 to 3653632 tid 1 2024-09-15T03:42:08.102 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.102 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.104 INFO:tasks.rados.rados.0.smithi105.stdout:356: writing smithi10536496-3 from 3653632 to 4079616 tid 2 2024-09-15T03:42:08.108 INFO:tasks.rados.rados.0.smithi105.stdout:356: writing smithi10536496-3 from 4079616 to 4800512 tid 3 2024-09-15T03:42:08.111 INFO:tasks.rados.rados.0.smithi105.stdout:356: writing smithi10536496-3 from 4800512 to 5210112 tid 4 2024-09-15T03:42:08.111 INFO:tasks.rados.rados.0.smithi105.stdout:357: delete oid 11 current snap is 34 2024-09-15T03:42:08.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:42:08.115 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing write tid 1 to smithi10536496-3 2024-09-15T03:42:08.116 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing write tid 2 to smithi10536496-3 2024-09-15T03:42:08.120 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing write tid 3 to smithi10536496-3 2024-09-15T03:42:08.120 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing write tid 4 to smithi10536496-3 2024-09-15T03:42:08.120 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing write tid 5 to smithi10536496-3 2024-09-15T03:42:08.122 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing write tid 6 to smithi10536496-3 2024-09-15T03:42:08.122 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 22 (ObjNum 140 snap 34 seq_num 140) dirty exists 2024-09-15T03:42:08.122 INFO:tasks.rados.rados.0.smithi105.stdout:356: left oid 3 (ObjNum 140 snap 34 seq_num 140) 2024-09-15T03:42:08.122 INFO:tasks.rados.rados.0.smithi105.stdout:356: done (1 left) 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:357: done (0 left) 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:358: read oid 37 snap -1 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:358: expect (ObjNum 116 snap 23 seq_num 116) 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:359: read oid 28 snap -1 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:359: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 38 current snap is 34 2024-09-15T03:42:08.123 INFO:tasks.rados.rados.0.smithi105.stdout:360: seq_num 142 ranges {0=442368,442368=720896,1163264=540672,1703936=688128} 2024-09-15T03:42:08.125 INFO:tasks.rados.rados.0.smithi105.stdout:360: writing smithi10536496-38 from 0 to 442368 tid 1 2024-09-15T03:42:08.129 INFO:tasks.rados.rados.0.smithi105.stdout:360: writing smithi10536496-38 from 442368 to 1163264 tid 2 2024-09-15T03:42:08.132 INFO:tasks.rados.rados.0.smithi105.stdout:360: writing smithi10536496-38 from 1163264 to 1703936 tid 3 2024-09-15T03:42:08.136 INFO:tasks.rados.rados.0.smithi105.stdout:360: writing smithi10536496-38 from 1703936 to 2392064 tid 4 2024-09-15T03:42:08.136 INFO:tasks.rados.rados.0.smithi105.stdout:361: setattr oid 14 current snap is 34 2024-09-15T03:42:08.137 INFO:tasks.rados.rados.0.smithi105.stdout:359: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing write tid 1 to smithi10536496-38 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing write tid 2 to smithi10536496-38 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing write tid 3 to smithi10536496-38 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing write tid 4 to smithi10536496-38 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing write tid 5 to smithi10536496-38 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing write tid 6 to smithi10536496-38 2024-09-15T03:42:08.150 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 61 (ObjNum 142 snap 34 seq_num 142) dirty exists 2024-09-15T03:42:08.151 INFO:tasks.rados.rados.0.smithi105.stdout:360: left oid 38 (ObjNum 142 snap 34 seq_num 142) 2024-09-15T03:42:08.151 INFO:tasks.rados.rados.0.smithi105.stdout:359: done (3 left) 2024-09-15T03:42:08.151 INFO:tasks.rados.rados.0.smithi105.stdout:360: done (2 left) 2024-09-15T03:42:08.151 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 41 current snap is 34 2024-09-15T03:42:08.151 INFO:tasks.rados.rados.0.smithi105.stdout:362: seq_num 143 ranges {0=458752,458752=475136,933888=655360,1589248=671744,2260992=147456} 2024-09-15T03:42:08.155 INFO:tasks.rados.rados.0.smithi105.stdout:362: writing smithi10536496-41 from 0 to 458752 tid 1 2024-09-15T03:42:08.157 INFO:tasks.rados.rados.0.smithi105.stdout:362: writing smithi10536496-41 from 458752 to 933888 tid 2 2024-09-15T03:42:08.161 INFO:tasks.rados.rados.0.smithi105.stdout:362: writing smithi10536496-41 from 933888 to 1589248 tid 3 2024-09-15T03:42:08.165 INFO:tasks.rados.rados.0.smithi105.stdout:362: writing smithi10536496-41 from 1589248 to 2260992 tid 4 2024-09-15T03:42:08.166 INFO:tasks.rados.rados.0.smithi105.stdout:362: writing smithi10536496-41 from 2260992 to 2408448 tid 5 2024-09-15T03:42:08.166 INFO:tasks.rados.rados.0.smithi105.stdout:363: read oid 40 snap -1 2024-09-15T03:42:08.167 INFO:tasks.rados.rados.0.smithi105.stdout:363: expect (ObjNum 122 snap 25 seq_num 122) 2024-09-15T03:42:08.167 INFO:tasks.rados.rados.0.smithi105.stdout:364: rmattr oid 9 current snap is 34 2024-09-15T03:42:08.167 INFO:tasks.rados.rados.0.smithi105.stdout:364: done (4 left) 2024-09-15T03:42:08.167 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 13 current snap is 34 2024-09-15T03:42:08.167 INFO:tasks.rados.rados.0.smithi105.stdout:365: seq_num 144 ranges {2146304=16384} 2024-09-15T03:42:08.179 INFO:tasks.rados.rados.0.smithi105.stdout:365: writing smithi10536496-13 from 2146304 to 2162688 tid 1 2024-09-15T03:42:08.179 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 17 current snap is 34 2024-09-15T03:42:08.180 INFO:tasks.rados.rados.0.smithi105.stdout:366: seq_num 145 ranges {4030464=32768} 2024-09-15T03:42:08.186 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:08.186+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:08.202 INFO:tasks.rados.rados.0.smithi105.stdout:366: writing smithi10536496-17 from 4030464 to 4063232 tid 1 2024-09-15T03:42:08.202 INFO:tasks.rados.rados.0.smithi105.stdout:367: rollback oid 42 current snap is 34 2024-09-15T03:42:08.202 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 42 to 29 2024-09-15T03:42:08.202 INFO:tasks.rados.rados.0.smithi105.stdout:358: expect (ObjNum 116 snap 23 seq_num 116) 2024-09-15T03:42:08.225 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.225 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.234 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:42:08.288 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:08.287+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:08.308 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 82 (ObjNum 137 snap 33 seq_num 137) dirty exists 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 1 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:358: done (6 left) 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:361: done (5 left) 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:368: delete oid 15 current snap is 34 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 2 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 3 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 4 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 5 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 6 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing write tid 7 to smithi10536496-41 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 94 (ObjNum 143 snap 34 seq_num 143) dirty exists 2024-09-15T03:42:08.309 INFO:tasks.rados.rados.0.smithi105.stdout:362: left oid 41 (ObjNum 143 snap 34 seq_num 143) 2024-09-15T03:42:08.310 INFO:tasks.rados.rados.0.smithi105.stdout:363: expect (ObjNum 122 snap 25 seq_num 122) 2024-09-15T03:42:08.374 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.374 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.383 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:42:08.424 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:08.423+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:08.454 INFO:tasks.rados.rados.0.smithi105.stdout:365: finishing write tid 1 to smithi10536496-13 2024-09-15T03:42:08.454 INFO:tasks.rados.rados.0.smithi105.stdout:365: finishing write tid 2 to smithi10536496-13 2024-09-15T03:42:08.454 INFO:tasks.rados.rados.0.smithi105.stdout:365: finishing write tid 3 to smithi10536496-13 2024-09-15T03:42:08.454 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 60 (ObjNum 144 snap 34 seq_num 144) dirty exists 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:365: left oid 13 (ObjNum 144 snap 34 seq_num 144) 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:367: finishing rollback tid 0 to smithi10536496-42 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:366: finishing write tid 1 to smithi10536496-17 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:367: finishing rollback tid 1 to smithi10536496-42 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:362: done (5 left) 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:363: done (4 left) 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:365: done (3 left) 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:368: done (2 left) 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 37 current snap is 34 2024-09-15T03:42:08.455 INFO:tasks.rados.rados.0.smithi105.stdout:369: seq_num 147 ranges {3227648=65536} 2024-09-15T03:42:08.471 INFO:tasks.rados.rados.0.smithi105.stdout:369: writing smithi10536496-37 from 3227648 to 3293184 tid 1 2024-09-15T03:42:08.472 INFO:tasks.rados.rados.0.smithi105.stdout:370: snap_create 2024-09-15T03:42:08.475 INFO:tasks.rados.rados.0.smithi105.stdout:369: finishing write tid 1 to smithi10536496-37 2024-09-15T03:42:08.479 INFO:tasks.rados.rados.0.smithi105.stdout:369: finishing write tid 2 to smithi10536496-37 2024-09-15T03:42:08.481 INFO:tasks.rados.rados.0.smithi105.stdout:369: finishing write tid 3 to smithi10536496-37 2024-09-15T03:42:08.481 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 85 (ObjNum 147 snap 34 seq_num 147) dirty exists 2024-09-15T03:42:08.482 INFO:tasks.rados.rados.0.smithi105.stdout:369: left oid 37 (ObjNum 147 snap 34 seq_num 147) 2024-09-15T03:42:08.489 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:08.488+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:08.522 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.522 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.530 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:42:08.615 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:08.613+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:08.667 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.667 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.675 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:42:08.687 INFO:tasks.rados.rados.0.smithi105.stdout:366: finishing write tid 2 to smithi10536496-17 2024-09-15T03:42:08.689 INFO:tasks.rados.rados.0.smithi105.stdout:366: finishing write tid 3 to smithi10536496-17 2024-09-15T03:42:08.689 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 49 (ObjNum 145 snap 34 seq_num 145) dirty exists 2024-09-15T03:42:08.689 INFO:tasks.rados.rados.0.smithi105.stdout:366: left oid 17 (ObjNum 145 snap 34 seq_num 145) 2024-09-15T03:42:08.690 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:08.690+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:08.782 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.783 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:42:08.792 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:08.792+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:08.853 INFO:tasks.rados.rados.0.smithi105.stdout:367: finishing rollback tid 2 to smithi10536496-42 2024-09-15T03:42:08.853 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 46 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-15T03:42:08.893 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:08.893+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:08.896 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:08.896 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:08.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-15T03:42:08.996 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:08.994+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:09.011 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-15T03:42:09.012 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.13 to 1 (10000) 2024-09-15T03:42:09.012 INFO:tasks.rados.rados.0.smithi105.stdout:366: done (3 left) 2024-09-15T03:42:09.013 INFO:tasks.rados.rados.0.smithi105.stdout:367: done (2 left) 2024-09-15T03:42:09.013 INFO:tasks.rados.rados.0.smithi105.stdout:369: done (1 left) 2024-09-15T03:42:09.013 INFO:tasks.rados.rados.0.smithi105.stdout:370: done (0 left) 2024-09-15T03:42:09.013 INFO:tasks.rados.rados.0.smithi105.stdout:371: snap_create 2024-09-15T03:42:09.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-15T03:42:09.054 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.055 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.064 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-15T03:42:09.096 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:09.095+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:09.196 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:09.196+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:09.234 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.234 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.242 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-15T03:42:09.326 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:09.326+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:09.387 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.387 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-15T03:42:09.398 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:09.398+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:09.499 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:09.498+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:09.538 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.538 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.546 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-15T03:42:09.600 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:09.600+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:09.649 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.649 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-15T03:42:09.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:42:09.701 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:09.701+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:09.761 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.761 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-15T03:42:09.802 INFO:tasks.ceph.osd.11.smithi105.stderr:2024-09-15T03:42:09.801+0000 7feedf1f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33109) UID: 0 2024-09-15T03:42:09.829 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:42:09.829 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:42:09.851 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:42:09.851 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:42:09.871 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:42:09.871 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:42:09.872 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.873 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.882 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-15T03:42:09.892 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:42:09.892 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:42:09.903 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:09.903+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:09.911 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:42:09.911 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:42:09.932 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:42:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:42:09.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:42:09.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:42:09.972 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:42:09.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:42:09.984 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:09.984 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:09.991 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:42:09.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:42:10.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-15T03:42:10.005 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:10.004+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:10.012 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:42:10.012 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:42:10.013 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.14 to 1 (10000) 2024-09-15T03:42:10.014 INFO:tasks.rados.rados.0.smithi105.stdout:371: done (0 left) 2024-09-15T03:42:10.015 INFO:tasks.rados.rados.0.smithi105.stdout:372: read oid 16 snap -1 2024-09-15T03:42:10.015 INFO:tasks.rados.rados.0.smithi105.stdout:372: expect deleted 2024-09-15T03:42:10.015 INFO:tasks.rados.rados.0.smithi105.stdout:373: rmattr oid 44 current snap is 36 2024-09-15T03:42:10.015 INFO:tasks.rados.rados.0.smithi105.stdout:374: copy_from oid 24 from oid 37 current snap is 36 2024-09-15T03:42:10.015 INFO:tasks.rados.rados.0.smithi105.stdout:375: copy_from oid 50 from oid 23 current snap is 36 2024-09-15T03:42:10.015 INFO:tasks.rados.rados.0.smithi105.stdout:376: setattr oid 1 current snap is 36 2024-09-15T03:42:10.017 INFO:tasks.rados.rados.0.smithi105.stdout:372: done (4 left) 2024-09-15T03:42:10.017 INFO:tasks.rados.rados.0.smithi105.stdout:377: snap_create 2024-09-15T03:42:10.022 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 66 (ObjNum 89 snap 14 seq_num 89) dirty exists 2024-09-15T03:42:10.027 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 90 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-15T03:42:10.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-15T03:42:10.029 INFO:tasks.rados.rados.0.smithi105.stdout:375: finishing copy_from racing read to smithi10536496-50 2024-09-15T03:42:10.032 INFO:tasks.rados.rados.0.smithi105.stdout:375: finishing copy_from to smithi10536496-50 2024-09-15T03:42:10.032 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 4 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:42:10.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:42:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:42:10.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:42:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:42:10.078 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:10.100 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:42:10.105 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:10.104+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:10.104+0000 7f0e4d0f8640 0 -- 172.21.15.5:0/1503902421 >> [v2:172.21.15.78:6808/32812,v1:172.21.15.78:6810/32812] conn(0x7f0e48195270 msgr2=0x7f0e48197650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:10.121 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:42:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:42:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-15T03:42:10.122 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:42:10.125 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:10.125 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:42:10.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-15T03:42:10.145 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:10.146 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:10.207 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:10.206+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:10.239 INFO:tasks.rados.rados.0.smithi105.stdout:374: finishing copy_from racing read to smithi10536496-24 2024-09-15T03:42:10.244 INFO:tasks.rados.rados.0.smithi105.stdout:374: finishing copy_from to smithi10536496-24 2024-09-15T03:42:10.244 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 85 (ObjNum 147 snap 34 seq_num 147) dirty exists 2024-09-15T03:42:10.255 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:42:10.255 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-15T03:42:10.255 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-09-15T03:42:10.256 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-09-15T03:42:10.256 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:42:10.260 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-15T03:42:10.308 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-09-15T03:42:10.307+0000 7f0dd8bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32802) UID: 0 2024-09-15T03:42:10.374 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:42:10.374 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-09-15T03:42:10.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=10, reqid=client.4562.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 68.300086242000006, 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084043045999999996, 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070944+0000", 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070945+0000", 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.124e-06 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.070959+0000", 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4212999999999999e-05 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154508+0000", 2024-09-15T03:42:10.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083548836000000001 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154515+0000", 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2080000000000003e-06 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154566+0000", 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1025e-05 2024-09-15T03:42:10.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154578+0000", 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154965+0000", 2024-09-15T03:42:10.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038690300000000002 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154987+0000", 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2172999999999999e-05 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=12, reqid=client.4562.0:150, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 68.216785982999994, 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035676890000000002, 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154244+0000", 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154245+0000", 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0699999999999996e-07 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.154487+0000", 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000242308 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156728+0000", 2024-09-15T03:42:10.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022401589999999998 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156736+0000", 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3259999999999996e-06 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156781+0000", 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4583000000000003e-05 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.156789+0000", 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8359999999999998e-06 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157761+0000", 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097187399999999996 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157812+0000", 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0695999999999997e-05 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=14, reqid=client.4562.0:152, at_version=35'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 68.213842092999997, 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028494564, 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157188+0000", 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157189+0000", 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3399999999999998e-07 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.157206+0000", 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7207e-05 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185008+0000", 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027801691999999999 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185015+0000", 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4250000000000001e-06 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185068+0000", 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2333000000000002e-05 2024-09-15T03:42:10.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185077+0000", 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0259999999999996e-06 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185660+0000", 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058343299999999996 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.185683+0000", 2024-09-15T03:42:10.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714e-05 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=17, reqid=client.4562.0:166, at_version=35'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 68.184145341999994, 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099130431000000005, 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.392 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186885+0000", 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186886+0000", 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9599999999999999e-07 2024-09-15T03:42:10.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.186906+0000", 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0482000000000001e-05 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285393+0000", 2024-09-15T03:42:10.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098487125999999994 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285401+0000", 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3780000000000003e-06 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285446+0000", 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028999999999999e-05 2024-09-15T03:42:10.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285453+0000", 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3119999999999996e-06 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.285995+0000", 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00054192900000000002 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:02.286015+0000", 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0579000000000001e-05 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=55, reqid=client.4562.0:269, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:42:10.398 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 59.349517157000001, 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030110250000000001, 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.021513+0000", 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8299999999999998e-07 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022295+0000", 2024-09-15T03:42:10.400 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078131100000000003 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022331+0000", 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6103000000000003e-05 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022338+0000", 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6410000000000002e-06 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022426+0000", 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7366000000000003e-05 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.022437+0000", 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.131e-05 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024502+0000", 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002065271 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:11.024524+0000", 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1739999999999999e-05 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 47/32 ECSubWrite(tid=66, reqid=osd.11.0:66, at_version=47'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 56.362499233999998, 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003545678, 2024-09-15T03:42:10.404 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008531+0000", 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.29e-07 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008562+0000", 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0871999999999999e-05 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.406 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008606+0000", 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4057999999999997e-05 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008618+0000", 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878999999999999e-05 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008722+0000", 2024-09-15T03:42:10.407 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104236 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.008732+0000", 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0142999999999999e-05 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012049+0000", 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033168920000000001 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:14.012077+0000", 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7368999999999998e-05 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/32 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=51'24, trim_to=0'0, roll_forward_to=50'22))", 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 52.346490101000001, 2024-09-15T03:42:10.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042098190000000001, 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:42:10.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024540+0000", 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9600000000000001e-07 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024564+0000", 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3714e-05 2024-09-15T03:42:10.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024603+0000", 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9048000000000003e-05 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024610+0000", 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1520000000000003e-06 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.412 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024717+0000", 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000106872 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.024728+0000", 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.096e-05 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028684+0000", 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039562190000000004 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:18.028750+0000", 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5658000000000001e-05 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.cs5 56/32 ECSubWrite(tid=88, reqid=osd.11.0:88, at_version=56'26, trim_to=0'0, roll_forward_to=51'24))", 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:42:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.307811852999997, 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003747588, 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063218+0000", 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063219+0000", 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1399999999999998e-07 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063262+0000", 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3173000000000002e-05 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063457+0000", 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019513700000000001 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063464+0000", 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6120000000000004e-06 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063517+0000", 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3477999999999999e-05 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.063528+0000", 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.112e-05 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066940+0000", 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034120299999999999 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:23.066966+0000", 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5724e-05 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.419 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 77/71 ECSubWrite(tid=149, reqid=osd.4.0:149, at_version=77'31, trim_to=0'0, roll_forward_to=75'29))", 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.348688807999999, 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021045639000000001, 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.420 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.421 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022341+0000", 2024-09-15T03:42:10.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.422 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022342+0000", 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0200000000000003e-07 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.022376+0000", 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4474999999999998e-05 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042674+0000", 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020297453 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042677+0000", 2024-09-15T03:42:10.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5939999999999998e-06 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042725+0000", 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7528000000000003e-05 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.042736+0000", 2024-09-15T03:42:10.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1420000000000001e-05 2024-09-15T03:42:10.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.043360+0000", 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00062334599999999999 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.043387+0000", 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7421000000000001e-05 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.427 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(osd.5.22:4 3.5s0 3:a7103f89:::smithi10536496-39:head [assert-version v41,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.329530196, 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097123769999999995, 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:42:10.428 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041500+0000", 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041501+0000", 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.218e-06 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041508+0000", 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.421e-06 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041525+0000", 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7164e-05 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041531+0000", 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9279999999999998e-06 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041589+0000", 2024-09-15T03:42:10.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8352e-05 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.041750+0000", 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000160657 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.051212+0000", 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094626369999999994 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.432 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpReadReply(3.5s0 77/72 ECSubReadReply(tid=4, attrs_read=0))", 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.323826773, 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034924600000000002, 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:42:10.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047203+0000", 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.047204+0000", 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4000000000000001e-07 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.434 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050148+0000", 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029437109999999999 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050597+0000", 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00044960600000000002 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050605+0000", 2024-09-15T03:42:10.435 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.695e-06 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050651+0000", 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5874000000000002e-05 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:44.050696+0000", 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4934e-05 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.436 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.es6 78/71 ECSubWrite(tid=92, reqid=osd.2.0:92, at_version=78'63, trim_to=0'0, roll_forward_to=77'59))", 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.339793063999998, 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003315836, 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:42:10.437 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031237+0000", 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031238+0000", 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0669999999999999e-06 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031262+0000", 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4187999999999999e-05 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031302+0000", 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9521999999999999e-05 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.031309+0000", 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0489999999999998e-06 2024-09-15T03:42:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032119+0000", 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00080987699999999997 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032128+0000", 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2089999999999994e-06 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.034523+0000", 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023950349999999998 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.034553+0000", 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9889000000000001e-05 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWrite(3.bs3 78/71 ECSubWrite(tid=151, reqid=osd.4.0:151, at_version=78'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.338815003000001, 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043349979999999996, 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032215+0000", 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.35e-07 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032236+0000", 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0922000000000001e-05 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032321+0000", 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4445999999999998e-05 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032327+0000", 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.516e-06 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032717+0000", 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038967199999999998 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.032727+0000", 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.876e-06 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036523+0000", 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037960239999999998 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:41:45.036550+0000", 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7307000000000001e-05 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:790 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~622592] snapc 38=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.022512+0000", 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3485184330000006, 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015895922, 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 790 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.022512+0000", 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.022512+0000", 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.022514+0000", 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.305e-06 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.023456+0000", 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00094207299999999998 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.023484+0000", 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7492e-05 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.023495+0000", 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1477e-05 2024-09-15T03:42:10.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.023577+0000", 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1322000000000005e-05 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op must wait for map", 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.023588+0000", 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1401e-05 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.026503+0000", 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029151440000000002 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.026684+0000", 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000180915 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.453 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.028256+0000", 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001571628 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033308+0000", 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050521139999999999 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.038319+0000", 2024-09-15T03:42:10.454 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050112660000000003 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.038408+0000", 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8784999999999998e-05 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.455 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:791 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~655360] snapc 38=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.029277+0000", 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3417530410000005, 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099397830000000006, 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:10.456 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 791 2024-09-15T03:42:10.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.457 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.462 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.029277+0000", 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.029277+0000", 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.029279+0000", 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.249e-06 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030512+0000", 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012326279999999999 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030532+0000", 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0387999999999999e-05 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030541+0000", 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8910000000000001e-06 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.465 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030609+0000", 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7173000000000002e-05 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030790+0000", 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000181378 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032129+0000", 2024-09-15T03:42:10.467 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001338667 2024-09-15T03:42:10.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.467 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.467 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.036711+0000", 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045828099999999997 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.039179+0000", 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024677240000000001 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.468 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.039217+0000", 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7874999999999999e-05 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 95/82 ECSubWriteReply(tid=19, last_complete=95'52, committed=1, applied=1))", 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.030276+0000", 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3407540210000004, 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003032982, 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.469 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030276+0000", 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030276+0000", 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030277+0000", 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.139e-06 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030301+0000", 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3847999999999999e-05 2024-09-15T03:42:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030633+0000", 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00033137800000000001 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.030640+0000", 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4270000000000002e-06 2024-09-15T03:42:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033292+0000", 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026515950000000001 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033309+0000", 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7595e-05 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:792 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~688128] snapc 38=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.031612+0000", 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3394178659999998, 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0078567340000000006, 2024-09-15T03:42:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 792 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.031612+0000", 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.031612+0000", 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.031632+0000", 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9528e-05 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032768+0000", 2024-09-15T03:42:10.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001136511 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032783+0000", 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4963000000000001e-05 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032791+0000", 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4549999999999998e-06 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033485+0000", 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00069366499999999999 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033596+0000", 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000111875 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.034525+0000", 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00092832199999999998 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.036807+0000", 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022826410000000002 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.039428+0000", 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026209969999999999 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.039469+0000", 2024-09-15T03:42:10.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0776999999999999e-05 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:793 3.5s0 3:a7103f89:::smithi10536496-39:head [append 0~212992] snapc 38=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.032809+0000", 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3382211690000005, 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068314990000000004, 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:10.480 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 793 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032809+0000", 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032809+0000", 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.032810+0000", 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3570000000000001e-06 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033057+0000", 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024641799999999998 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033073+0000", 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6325999999999999e-05 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033081+0000", 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9719999999999992e-06 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.035299+0000", 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022182370000000001 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.035377+0000", 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7131000000000004e-05 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.036158+0000", 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078111599999999999 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.037931+0000", 2024-09-15T03:42:10.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017734140000000001 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.039555+0000", 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016235500000000001 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.039641+0000", 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5977999999999996e-05 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.485 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:794 3.5s0 3:a7103f89:::smithi10536496-39:head [setxattr _header (57) in=7b] snapc 38=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.033089+0000", 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3379416590000002, 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.274050034, 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 794 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.486 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033089+0000", 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033089+0000", 2024-09-15T03:42:10.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033089+0000", 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5199999999999996e-07 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033093+0000", 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6069999999999999e-06 2024-09-15T03:42:10.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033101+0000", 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6699999999999994e-06 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033117+0000", 2024-09-15T03:42:10.489 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6776e-05 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.037077+0000", 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039596639999999999 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.037188+0000", 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000110888 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.037530+0000", 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00034207699999999998 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.038559+0000", 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00102871 2024-09-15T03:42:10.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.307039+0000", 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.26848034199999998 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.307139+0000", 2024-09-15T03:42:10.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9548000000000003e-05 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.4562.0:795 3.5s0 3:a7103f89:::smithi10536496-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-09-15T03:42:02.033123+0000", 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.3379074190000004, 2024-09-15T03:42:10.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.275989549, 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.4562", 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 795 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-09-15T03:42:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033123+0000", 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033123+0000", 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-09-15T03:42:10.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033123+0000", 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2499999999999995e-07 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033128+0000", 2024-09-15T03:42:10.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.51e-06 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033138+0000", 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0404000000000001e-05 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.497 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.033143+0000", 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5109999999999997e-06 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.037807+0000", 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046645999999999996 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.037922+0000", 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011411700000000001 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.307559+0000", 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.26963748900000001 2024-09-15T03:42:10.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.307655+0000", 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5736000000000006e-05 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-15T03:42:10.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-09-15T03:42:02.309112+0000", 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001457657 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-15T03:42:10.501 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:42:10.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:42:10.503 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:10.448+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:10.549 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:10.548+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:10.617 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:10.618 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:42:10.618 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:42:10.618 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:10.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-15T03:42:10.650 INFO:tasks.ceph.osd.10.smithi078.stderr:2024-09-15T03:42:10.649+0000 7fde0f5f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32805) UID: 0 2024-09-15T03:42:10.751 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:10.750+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:10.771 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:10.771 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:42:10.771 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:42:10.772 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:42:10.772 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:10.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-15T03:42:10.853 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:42:10.852+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:42:10.883 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=1, reqid=client.4562.0:1, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.864171846999994, 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094904819999999997, 2024-09-15T03:42:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:42:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017498+0000", 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.23e-07 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017537+0000", 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8838000000000003e-05 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017912+0000", 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037478400000000002 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017919+0000", 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8010000000000004e-06 2024-09-15T03:42:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017987+0000", 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7138000000000007e-05 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.017998+0000", 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0908e-05 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026958+0000", 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089603229999999992 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.026988+0000", 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0467000000000002e-05 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:25, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.860061415000004, 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010778574000000001, 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021608+0000", 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021609+0000", 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021692+0000", 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2269999999999997e-05 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021735+0000", 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3581000000000002e-05 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021743+0000", 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3119999999999994e-06 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021822+0000", 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8437999999999997e-05 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.021830+0000", 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2989999999999997e-06 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032339+0000", 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010508768999999999 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032387+0000", 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7851e-05 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=0, reqid=client.4562.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.859408630999994, 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010171065999999999, 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022261+0000", 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022262+0000", 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.043e-06 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022318+0000", 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6208000000000001e-05 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022353+0000", 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4437000000000002e-05 2024-09-15T03:42:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022362+0000", 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1940000000000004e-06 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022408+0000", 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5775000000000001e-05 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022419+0000", 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1450999999999999e-05 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032351+0000", 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099319260000000006 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032432+0000", 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1032000000000006e-05 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 34/32 ECSubWrite(tid=4, reqid=client.4562.0:23, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.858741131000002, 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094003279999999995, 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022929+0000", 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022930+0000", 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7999999999999993e-07 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022949+0000", 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9916e-05 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.022995+0000", 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5173000000000003e-05 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023004+0000", 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2990000000000002e-06 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023067+0000", 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023073+0000", 2024-09-15T03:42:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032300+0000", 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092268320000000008 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032329+0000", 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8779999999999999e-05 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 34/32 ECSubWrite(tid=1, reqid=client.4562.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.858005425000002, 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087825990000000003, 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:42:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023664+0000", 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023665+0000", 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5199999999999995e-07 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023681+0000", 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-05 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023713+0000", 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1726000000000001e-05 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023720+0000", 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6830000000000002e-06 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023765+0000", 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4988999999999999e-05 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.023771+0000", 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9109999999999998e-06 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032437+0000", 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086659760000000006 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032447+0000", 2024-09-15T03:42:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0222e-05 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854529081999999, 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013181325000000001, 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027141+0000", 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7000000000000004e-07 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027155+0000", 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3424000000000001e-05 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027190+0000", 2024-09-15T03:42:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5738000000000002e-05 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027199+0000", 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1999999999999994e-06 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027257+0000", 2024-09-15T03:42:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8690000000000002e-05 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027265+0000", 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8860000000000004e-06 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040256+0000", 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012990323999999999 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040322+0000", 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6392999999999999e-05 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854050377999997, 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012779128000000001, 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027619+0000", 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027620+0000", 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.32e-07 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027692+0000", 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000006e-05 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027773+0000", 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0605000000000005e-05 2024-09-15T03:42:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027782+0000", 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027884+0000", 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010173100000000001 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.027894+0000", 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9210000000000004e-06 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040357+0000", 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463284999999999 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040398+0000", 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1026000000000003e-05 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=3, reqid=client.4562.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:42:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.852762674000004, 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011382537, 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028907+0000", 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028908+0000", 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0300000000000001e-06 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028922+0000", 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4329e-05 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028975+0000", 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2573e-05 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028983+0000", 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029027+0000", 2024-09-15T03:42:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3912000000000002e-05 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029036+0000", 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9560000000000003e-06 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040254+0000", 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011217632999999999 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040290+0000", 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5880999999999997e-05 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=3, reqid=client.4562.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.852749940999999, 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011534467, 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028920+0000", 2024-09-15T03:42:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028921+0000", 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.057e-06 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028959+0000", 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8217999999999999e-05 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.028993+0000", 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4425999999999999e-05 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029011+0000", 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7465e-05 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029270+0000", 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025923899999999998 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029280+0000", 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040431+0000", 2024-09-15T03:42:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011151283 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040454+0000", 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3309e-05 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=8, reqid=client.4562.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:42:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.852111879999995, 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013707001, 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029558+0000", 2024-09-15T03:42:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029559+0000", 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.054e-06 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029676+0000", 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117273 2024-09-15T03:42:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029720+0000", 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4026000000000002e-05 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029728+0000", 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3240000000000003e-06 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030022+0000", 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029338200000000002 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030037+0000", 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5082000000000001e-05 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043234+0000", 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013196548000000001 2024-09-15T03:42:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043265+0000", 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1312000000000001e-05 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=2, reqid=client.4562.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:42:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.851051440000006, 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097244270000000008, 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030618+0000", 2024-09-15T03:42:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030619+0000", 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999998e-07 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030630+0000", 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030662+0000", 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1813000000000002e-05 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030680+0000", 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8306999999999999e-05 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030722+0000", 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1823000000000003e-05 2024-09-15T03:42:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.030729+0000", 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2049999999999996e-06 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040325+0000", 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095952650000000004 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.040343+0000", 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8079000000000001e-05 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.850059938000001, 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011724098000000001, 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:42:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031610+0000", 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031742+0000", 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013129600000000001 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031779+0000", 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7311000000000002e-05 2024-09-15T03:42:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031787+0000", 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031842+0000", 2024-09-15T03:42:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4617999999999998e-05 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031856+0000", 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.42e-05 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043313+0000", 2024-09-15T03:42:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011457292000000001 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043334+0000", 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=9, reqid=client.4562.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.849861496000003, 2024-09-15T03:42:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011489871, 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031808+0000", 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031809+0000", 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0130000000000001e-06 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031830+0000", 2024-09-15T03:42:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0801999999999998e-05 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031864+0000", 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3575000000000003e-05 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031869+0000", 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5029999999999997e-06 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032118+0000", 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024897299999999999 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032128+0000", 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4879999999999996e-06 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043276+0000", 2024-09-15T03:42:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01114878 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043298+0000", 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1736999999999999e-05 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as2 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.849190883000006, 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010878669000000001, 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032479+0000", 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032507+0000", 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7140000000000001e-05 2024-09-15T03:42:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032542+0000", 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4986e-05 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032549+0000", 2024-09-15T03:42:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2130000000000002e-06 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034852+0000", 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023031089999999998 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.034862+0000", 2024-09-15T03:42:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0156000000000001e-05 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043343+0000", 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084812489999999997 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043357+0000", 2024-09-15T03:42:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4219000000000001e-05 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 35/32 ECSubWrite(tid=20, reqid=client.4562.0:159, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:42:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.721251244000001, 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091519410000000002, 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160418+0000", 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160419+0000", 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7499999999999999e-07 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160707+0000", 2024-09-15T03:42:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028803500000000002 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160722+0000", 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5235000000000001e-05 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160727+0000", 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7600000000000002e-06 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160775+0000", 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7732999999999999e-05 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.160785+0000", 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0380999999999999e-05 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169551+0000", 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087657610000000004 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.169570+0000", 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9261000000000001e-05 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "replica_scrub(pg: 3.2s5,from:0'0,to:35'2,epoch:76/51,start:3:40000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:43.748393+0000", 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.133277066000002, 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82562717100000005, 2024-09-15T03:42:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748393+0000", 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748393+0000", 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748394+0000", 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7100000000000011e-07 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:43.748415+0000", 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573901+0000", 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82548617499999999 2024-09-15T03:42:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573909+0000", 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7589999999999997e-06 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573979+0000", 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0051999999999999e-05 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.573992+0000", 2024-09-15T03:42:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2366e-05 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.574020+0000", 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8140000000000002e-05 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 88/44 ECSubWrite(tid=149, reqid=client.4562.0:746, at_version=88'60, trim_to=0'0, roll_forward_to=87'58))", 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:55.049515+0000", 2024-09-15T03:42:10.969 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.832154299000001, 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0123047, 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049515+0000", 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049515+0000", 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049516+0000", 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-07 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049680+0000", 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164473 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049746+0000", 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6353000000000007e-05 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049756+0000", 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049889+0000", 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013360800000000001 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049902+0000", 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2724999999999999e-05 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.061795+0000", 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011892962 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.061820+0000", 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4919999999999999e-05 2024-09-15T03:42:10.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.8s3 88/44 ECSubWrite(tid=150, reqid=client.4562.0:747, at_version=88'61, trim_to=0'0, roll_forward_to=87'58))", 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:55.051514+0000", 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.830155923, 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010382472, 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051514+0000", 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051514+0000", 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051515+0000", 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.19e-06 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051686+0000", 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171514 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051726+0000", 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9563000000000001e-05 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051733+0000", 2024-09-15T03:42:10.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7789999999999998e-06 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.052593+0000", 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086055599999999995 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.052605+0000", 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1895e-05 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.061870+0000", 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092643159999999999 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.061896+0000", 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6659e-05 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es2 94/71 ECSubWrite(tid=117, reqid=osd.2.0:117, at_version=94'83, trim_to=0'0, roll_forward_to=93'79))", 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:01.024462+0000", 2024-09-15T03:42:10.978 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8572077319999991, 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62336351000000001, 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024462+0000", 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024462+0000", 2024-09-15T03:42:10.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024462+0000", 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000002e-07 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024488+0000", 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5528999999999999e-05 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.646936+0000", 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62244865000000005 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.646948+0000", 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1676e-05 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.647052+0000", 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103408 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.647059+0000", 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2409999999999998e-06 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.647799+0000", 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073990899999999997 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.647825+0000", 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6809999999999999e-05 2024-09-15T03:42:10.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.17s5 94/51 ECSubWrite(tid=167, reqid=osd.5.0:167, at_version=94'66, trim_to=0'0, roll_forward_to=91'62))", 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:01.024900+0000", 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8567695999999998, 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010032231000000001, 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:10.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024900+0000", 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024900+0000", 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024900+0000", 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7100000000000001e-07 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.024962+0000", 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1976000000000004e-05 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.025333+0000", 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037099600000000001 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.025337+0000", 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1189999999999999e-06 2024-09-15T03:42:10.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.027632+0000", 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022946870000000001 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.027642+0000", 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0451999999999999e-05 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.034906+0000", 2024-09-15T03:42:10.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072636360000000004 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.034932+0000", 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6194e-05 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:10.987 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:10.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-15T03:42:10.989 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:10.988+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:11.017 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.15 to 1 (10000) 2024-09-15T03:42:11.018 INFO:tasks.rados.rados.0.smithi105.stdout:373: done (4 left) 2024-09-15T03:42:11.018 INFO:tasks.rados.rados.0.smithi105.stdout:374: done (3 left) 2024-09-15T03:42:11.018 INFO:tasks.rados.rados.0.smithi105.stdout:375: done (2 left) 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:376: done (1 left) 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:377: done (0 left) 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:378: read oid 1 snap 33 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:378: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:379: copy_from oid 17 from oid 47 current snap is 37 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:380: rollback oid 2 current snap is 37 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 2 to 29 2024-09-15T03:42:11.019 INFO:tasks.rados.rados.0.smithi105.stdout:381: snap_create 2024-09-15T03:42:11.025 INFO:tasks.rados.rados.0.smithi105.stdout:380: finishing rollback tid 0 to smithi10536496-2 2024-09-15T03:42:11.025 INFO:tasks.rados.rados.0.smithi105.stdout:380: finishing rollback tid 1 to smithi10536496-2 2024-09-15T03:42:11.026 INFO:tasks.rados.rados.0.smithi105.stdout:380: finishing rollback tid 2 to smithi10536496-2 2024-09-15T03:42:11.026 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 92 (ObjNum 102 snap 16 seq_num 102) dirty exists 2024-09-15T03:42:11.029 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-15T03:42:11.034 INFO:tasks.rados.rados.0.smithi105.stdout:379: finishing copy_from racing read to smithi10536496-17 2024-09-15T03:42:11.036 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-15T03:42:11.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:11.043 INFO:tasks.rados.rados.0.smithi105.stdout:379: finishing copy_from to smithi10536496-17 2024-09-15T03:42:11.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 10 (ObjNum 61 snap 4 seq_num 61) dirty exists 2024-09-15T03:42:11.122 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:11.121+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:919 3.8s0 3:136df151:::smithi10536496-1:41 [read 0~32768,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104)", 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:11.018739+0000", 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.114459822, 2024-09-15T03:42:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114481308, 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 919 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018739+0000", 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018739+0000", 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018740+0000", 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1800000000000002e-07 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018742+0000", 2024-09-15T03:42:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8279999999999999e-06 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018745+0000", 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018749+0000", 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.579e-06 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018860+0000", 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011106 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018866+0000", 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5879999999999997e-06 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.021466+0000", 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026009459999999998 2024-09-15T03:42:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.021572+0000", 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105473 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-15T03:42:11.138 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:11.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-15T03:42:11.190 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:11.190+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:11.244 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:11.244 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:42:11.244 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:42:11.244 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:42:11.244 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:11.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-15T03:42:11.291 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:11.290+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:3 3.8s0 3:136df151:::smithi10536496-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-15T03:42:11.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.337271569999999, 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016082863999999999, 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016638+0000", 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016639+0000", 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8500000000000002e-07 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016640+0000", 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2729999999999999e-06 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016645+0000", 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.113e-06 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.016650+0000", 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3489999999999999e-06 2024-09-15T03:42:11.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019489+0000", 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028386459999999998 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.019567+0000", 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7417999999999996e-05 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029120+0000", 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095539030000000007 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.029205+0000", 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4301000000000005e-05 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032721+0000", 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003516576 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.361 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:63 3.8s0 3:12b3a6c7:::smithi10536496-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.310177436000004, 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012980953, 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.362 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 63 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043733+0000", 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.72e-07 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043735+0000", 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8279999999999999e-06 2024-09-15T03:42:11.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043740+0000", 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9110000000000001e-06 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.043742+0000", 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6460000000000002e-06 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045685+0000", 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001942706 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045767+0000", 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2405999999999997e-05 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054190+0000", 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084225670000000006 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.054302+0000", 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112184 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.056714+0000", 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024113329999999999 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.367 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:286 3.8s0 3:136df151:::smithi10536496-1:head [create,setxattr dj[s][nsgcdlaO`ntnnogS`sZdZO[V\\]llWi\\gjj (11296) in=40b,setxattr qRNaVOjbUaQsk\\XsiZbOhrsOSoqX_Tqndtkb\\_Sa (1297) in=40b,setxattr `hdQPrsh[aUo`rPcbh\\tWT\\PatkS^p[P_Pdmcgdi (840) in=40b,setxattr RXicf\\^UcqO_mXh^am]mfcdttd\\mfOiPqZcomOo` (1822) in=40b,setxattr oZlXqWbdtOrc_U_m[apaV_Nr[hOOUgj`sVNpq^Rp (1473) in=40b,setxattr TrdTbcloOaiWXhjih\\mdXfYZcg\\^aOrnVOj^sf`s (10104) in=40b,setxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt (1263) in=40b,setxattr ^qaVseQnk[`qOXVa`m]]NtQ\\psXaNYand^tQYrWt (1051) in=40b,setxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q (418) in=40b,setxattr cfoPTk\\o\\Qqbi^XNOW[dbpcRb`OledY]YqcmmqUr (19016) in=40b,setxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd (10104) in=35b,setxattr mtVO^NRpeT]rQZYdTadneUO_^VXn^UdhZ^\\bUQiR (769) in=40b,setxattr prahkiNnm`U[rXSatop\\lk_p\\WVkdatg[XURZrsm (4560) in=40b,setxattr isZNUepUtPtRnYQaXpfelSVVR\\apSohRfqcfV`T_ (10033) in=40b,setxattr QZkproN[Zssrnra[rrbkaSXpatcreQpt`tWbOdoO (8349) in=40b,setxattr YqVPkaeiY^Tbe\\Rr\\heXkWjtOsi`iaN[TO\\^kP^] (876) in=40b,setxattr ZdfcZ]]snWVj]dZhZ]YSjVneRRoNbk`d^een^nqN (3368) in=40b,setxattr Y^XWcNfadqaYlbNpPbVRoPtgWreNtPrhWlfbbYY] (10804) in=40b,setxattr a`l_j\\^WlXY[ekSUURUhO (10104) in=21b,setxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ (1541) in=40b] snapc b=[b,a] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.32567667, 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025921639, 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 286 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028233+0000", 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028235+0000", 2024-09-15T03:42:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.872e-06 2024-09-15T03:42:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028332+0000", 2024-09-15T03:42:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-09-15T03:42:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028352+0000", 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0262999999999999e-05 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.028359+0000", 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4390000000000003e-06 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029687+0000", 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013271909999999999 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.029881+0000", 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000194792 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.031152+0000", 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012703910000000001 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.033779+0000", 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002626724 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054105+0000", 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020326455 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.054155+0000", 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0021000000000001e-05 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.11s4 51/49 ECSubWrite(tid=10, reqid=osd.10.0:10, at_version=51'19, trim_to=0'0, roll_forward_to=50'17))", 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.328604370000001, 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013366444, 2024-09-15T03:42:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025306+0000", 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025307+0000", 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025333+0000", 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6886999999999999e-05 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025376+0000", 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2821000000000002e-05 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025384+0000", 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6520000000000006e-06 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025562+0000", 2024-09-15T03:42:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017756799999999999 2024-09-15T03:42:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025571+0000", 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-06 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038645+0000", 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013073817999999999 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.038672+0000", 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7131000000000002e-05 2024-09-15T03:42:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:438 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~753664] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.320202577000003, 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025886468999999999, 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 438 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033707+0000", 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.033710+0000", 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4380000000000002e-06 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034822+0000", 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011121359999999999 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034849+0000", 2024-09-15T03:42:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7406000000000002e-05 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034859+0000", 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6830000000000003e-06 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034943+0000", 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4303999999999998e-05 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.034953+0000", 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5019999999999998e-06 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039409+0000", 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044563520000000002 2024-09-15T03:42:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039523+0000", 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113514 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.041482+0000", 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019589410000000001 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048318+0000", 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006836033 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059547+0000", 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011228802 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059594+0000", 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7358000000000003e-05 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:439 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~720896] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.313931033000003, 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019746224, 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 439 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039979+0000", 2024-09-15T03:42:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.039981+0000", 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829999999999999e-06 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042101+0000", 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021201850000000001 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042116+0000", 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4994000000000001e-05 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042124+0000", 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7729999999999999e-06 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043391+0000", 2024-09-15T03:42:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012674959999999999 2024-09-15T03:42:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.043512+0000", 2024-09-15T03:42:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120469 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.044649+0000", 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001137327 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.048320+0000", 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036706320000000001 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059680+0000", 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011360093 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.059725+0000", 2024-09-15T03:42:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5472000000000001e-05 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:440 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~49152] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.311739971999998, 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02155733, 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 440 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042170+0000", 2024-09-15T03:42:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042171+0000", 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.39e-06 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042243+0000", 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1474e-05 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042256+0000", 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3016999999999999e-05 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042262+0000", 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4269999999999996e-06 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045435+0000", 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031722920000000002 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.045540+0000", 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105648 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046184+0000", 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064343199999999999 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049337+0000", 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003153089 2024-09-15T03:42:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063687+0000", 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01435036 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063727+0000", 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0200999999999997e-05 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:441 3.8s0 3:115f2b89:::smithi10536496-36:head [setxattr _header (57) in=7b] snapc 19=[19,18,17] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.311639540000002, 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021588698, 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 441 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042271+0000", 2024-09-15T03:42:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5499999999999997e-07 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042275+0000", 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7900000000000001e-06 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042284+0000", 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8440000000000004e-06 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042294+0000", 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6569999999999992e-06 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046742+0000", 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044484989999999999 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.046840+0000", 2024-09-15T03:42:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7651000000000004e-05 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047278+0000", 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043781499999999998 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.049368+0000", 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020900620000000002 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063805+0000", 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014437324 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.063859+0000", 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4301000000000001e-05 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:442 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.311611176, 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028656429000000001, 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 442 2024-09-15T03:42:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042299+0000", 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042302+0000", 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4990000000000001e-06 2024-09-15T03:42:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042309+0000", 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7139999999999996e-06 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.042313+0000", 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2660000000000003e-06 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047492+0000", 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051795030000000002 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.047573+0000", 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0895000000000004e-05 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064140+0000", 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016566658000000001 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.064227+0000", 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199000000000003e-05 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:29.070955+0000", 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067281440000000001 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:631 3.8s0 3:115f2b89:::smithi10536496-36:head [copy-from ver 41] snapc 28=[28,27,26,25] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:42:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.317729788000001, 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.46301878800000001, 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 631 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036180+0000", 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036182+0000", 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9750000000000001e-06 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036188+0000", 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8980000000000001e-06 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036204+0000", 2024-09-15T03:42:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5906e-05 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036211+0000", 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6490000000000002e-06 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036287+0000", 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5835999999999997e-05 2024-09-15T03:42:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036393+0000", 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106378 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.057720+0000", 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021326686000000001 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.060505+0000", 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027857680000000001 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.499151+0000", 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.43864593699999999 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.499199+0000", 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7754999999999999e-05 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:632 3.8s0 3:115f2b89:::smithi10536496-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:42:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.317683716000001, 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023198263, 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 632 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036226+0000", 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036227+0000", 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7199999999999998e-07 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036230+0000", 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7949999999999999e-06 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036238+0000", 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1829999999999994e-06 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036242+0000", 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740000000000004e-06 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036579+0000", 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033680299999999999 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for blocked object", 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.036635+0000", 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6545999999999997e-05 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059280+0000", 2024-09-15T03:42:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022644272 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059341+0000", 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1425000000000002e-05 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:44.059425+0000", 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3592999999999997e-05 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:746 3.8s0 3:12b3a6c7:::smithi10536496-21:head [append 0~540672] snapc 32=[32,2e,25] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:55.042482+0000", 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.311427627, 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019675714, 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 746 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.042482+0000", 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.042482+0000", 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.042484+0000", 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.694e-06 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.043158+0000", 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067387299999999996 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.043175+0000", 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6823000000000001e-05 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.043181+0000", 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2670000000000003e-06 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.043255+0000", 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3559000000000005e-05 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.043263+0000", 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0749999999999998e-06 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.048298+0000", 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005035454 2024-09-15T03:42:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.048410+0000", 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011225700000000001 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.049509+0000", 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010981229999999999 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.059046+0000", 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095371870000000008 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.062068+0000", 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003022435 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.062158+0000", 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9967000000000001e-05 2024-09-15T03:42:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:747 3.8s0 3:12b3a6c7:::smithi10536496-21:head [append 0~720896] snapc 32=[32,2e,25] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:55.046438+0000", 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.307471605, 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015924720999999999, 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 747 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.046438+0000", 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.046438+0000", 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.046440+0000", 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.161e-06 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.047171+0000", 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073156399999999998 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.047184+0000", 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.255e-05 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.047193+0000", 2024-09-15T03:42:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1989999999999995e-06 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.047239+0000", 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6520000000000002e-05 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.047244+0000", 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4429999999999996e-06 2024-09-15T03:42:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.050583+0000", 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033388039999999999 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.050659+0000", 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5859000000000007e-05 2024-09-15T03:42:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.051444+0000", 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078498000000000001 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.059047+0000", 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076035679999999998 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.062282+0000", 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003234649 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:55.062363+0000", 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1423999999999998e-05 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:821 3.17s0 3:edf86ef6:::smithi10536496-37:head [rollback 3d] snapc 3e=[3e,3d,3c] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:05.014666+0000", 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.3392441569999995, 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013162695, 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 821 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014666+0000", 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014666+0000", 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014666+0000", 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5499999999999998e-07 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014671+0000", 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014684+0000", 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563999999999999e-05 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014689+0000", 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9019999999999998e-06 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014766+0000", 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7114999999999996e-05 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014770+0000", 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2089999999999999e-06 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.017628+0000", 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028571260000000002 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.017719+0000", 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1683999999999998e-05 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.018222+0000", 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050238099999999999 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.027641+0000", 2024-09-15T03:42:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094196550000000007 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.027712+0000", 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0507999999999996e-05 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.027829+0000", 2024-09-15T03:42:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116767 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:822 3.17s0 3:edf86ef6:::smithi10536496-37:head [append 0~0] snapc 3e=[3e,3d,3c] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:05.014695+0000", 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.3392147799999998, 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014106843000000001, 2024-09-15T03:42:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 822 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014695+0000", 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014695+0000", 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014696+0000", 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9799999999999999e-07 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014698+0000", 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014706+0000", 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014711+0000", 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3329999999999998e-06 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014788+0000", 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7330000000000007e-05 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.014791+0000", 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5030000000000002e-06 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.018830+0000", 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004038982 2024-09-15T03:42:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.018934+0000", 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103304 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.019733+0000", 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079976999999999999 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.027832+0000", 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080984990000000003 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.027906+0000", 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4443999999999994e-05 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.028802+0000", 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089574499999999998 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:836 3.8s0 3:115f2b89:::smithi10536496-36:head [append 0~360448] snapc 3e=[3e,3d,3c] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:05.057464+0000", 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.2964460960000004, 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013143058000000001, 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 836 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057464+0000", 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057464+0000", 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057465+0000", 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1689999999999999e-06 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057824+0000", 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035875300000000001 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057834+0000", 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0183000000000001e-05 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057838+0000", 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5999999999999998e-06 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057881+0000", 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3201999999999997e-05 2024-09-15T03:42:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057994+0000", 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113213 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.058620+0000", 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062617900000000004 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.062895+0000", 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042751999999999998 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.070556+0000", 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076600690000000003 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.070607+0000", 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1490000000000003e-05 2024-09-15T03:42:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:837 3.8s0 3:115f2b89:::smithi10536496-36:head [setxattr _header (57) in=7b] snapc 3e=[3e,3d,3c] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:05.057845+0000", 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.2960652570000004, 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61185887299999997, 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 837 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057845+0000", 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057845+0000", 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057845+0000", 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8999999999999996e-07 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057849+0000", 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.151e-06 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057854+0000", 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.921e-06 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057856+0000", 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1610000000000001e-06 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.059192+0000", 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001336154 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.059285+0000", 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2761000000000002e-05 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.059676+0000", 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039158900000000002 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.062909+0000", 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032331460000000001 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.669642+0000", 2024-09-15T03:42:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60673235000000003 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.669704+0000", 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1950000000000001e-05 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:838 3.8s0 3:115f2b89:::smithi10536496-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:05.057893+0000", 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.2960168850000002, 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61342676100000004, 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 838 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057893+0000", 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057893+0000", 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057894+0000", 2024-09-15T03:42:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-15T03:42:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057897+0000", 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0960000000000001e-06 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057908+0000", 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1209000000000001e-05 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.057912+0000", 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2849999999999996e-06 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.059894+0000", 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019814469999999999 2024-09-15T03:42:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.059964+0000", 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9801999999999993e-05 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.670030+0000", 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61006643000000005 2024-09-15T03:42:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.670133+0000", 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102817 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:05.671320+0000", 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001187062 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:856 3.17s0 3:ec08d906:::smithi10536496-14:head [setxattr _header (57) in=7b] snapc 3f=[3f,3e,3d,3c] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:06.087148+0000", 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.266761926, 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.68025991299999999, 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 856 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087148+0000", 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087148+0000", 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087149+0000", 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999999e-07 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087153+0000", 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3560000000000003e-06 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087160+0000", 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7689999999999999e-06 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087164+0000", 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2610000000000004e-06 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.091329+0000", 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041650150000000002 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.091392+0000", 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2738999999999997e-05 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.091702+0000", 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000309959 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.095731+0000", 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040287179999999997 2024-09-15T03:42:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.767339+0000", 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.67160810599999998 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.767408+0000", 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9270000000000006e-05 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:857 3.17s0 3:ec08d906:::smithi10536496-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:06.087169+0000", 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2667410329999997, 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.68179734000000003, 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 857 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087169+0000", 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087169+0000", 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087170+0000", 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1800000000000005e-07 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087172+0000", 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0509999999999999e-06 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087176+0000", 2024-09-15T03:42:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-06 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.087179+0000", 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.554e-06 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.091985+0000", 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048064479999999996 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.092047+0000", 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1730999999999996e-05 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.767862+0000", 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.67581526999999997 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.767989+0000", 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012674499999999999 2024-09-15T03:42:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:06.768966+0000", 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097727300000000007 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.490 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:11.492 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:11.491+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:11.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-15T03:42:11.495 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:11.496 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:11.498 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-15T03:42:11.016048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89129523,"num_objects":58,"num_object_clones":13,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":3,"num_read":365,"num_read_kb":144775,"num_write":650,"num_write_kb":112082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":53411859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1171,"ondisk_log_size":1171,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":4373092,"kb_used_data":168464,"kb_used_omap":23,"kb_used_meta":4204392,"kb_avail":1316832668,"statfs":{"total":1352914698240,"available":1348436652032,"internally_reserved":0,"allocated":172507136,"data_stored":192779186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23856,"internal_metadata":4305298128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11878400,"num_objects":13,"num_object_clones":12,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":21,"num_read_kb":10424,"num_write":91,"num_write_kb":11600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.953528"},"pg_stats":[{"pgid":"3.19","version":"70'27","reported_seq":297,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.066089+0000","last_change":"2024-09-15T03:42:07.066744+0000","last_active":"2024-09-15T03:42:07.066089+0000","last_peered":"2024-09-15T03:42:07.066089+0000","last_clean":"2024-09-15T03:42:07.066089+0000","last_became_active":"2024-09-15T03:41:50.026750+0000","last_became_peered":"2024-09-15T03:41:50.026750+0000","last_unstale":"2024-09-15T03:42:07.066089+0000","last_undegraded":"2024-09-15T03:42:07.066089+0000","last_fullsized":"2024-09-15T03:42:07.066089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'21","last_scrub_stamp":"2024-09-15T03:41:30.737422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.737422+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:59.283558+0000","scrub_duration":0.0027345220000000001,"objects_trimmed":0,"snaptrim_duration":0.001624871,"stat_sum":{"num_bytes":4685824,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":18626,"num_write":2,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12,11,9,13,7,1],"acting":[8,12,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.18","version":"58'18","reported_seq":263,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071574+0000","last_change":"2024-09-15T03:42:07.072210+0000","last_active":"2024-09-15T03:42:07.071574+0000","last_peered":"2024-09-15T03:42:07.071574+0000","last_clean":"2024-09-15T03:42:07.071574+0000","last_became_active":"2024-09-15T03:41:50.021784+0000","last_became_peered":"2024-09-15T03:41:50.021784+0000","last_unstale":"2024-09-15T03:42:07.071574+0000","last_undegraded":"2024-09-15T03:42:07.071574+0000","last_fullsized":"2024-09-15T03:42:07.071574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"43'9","last_scrub_stamp":"2024-09-15T03:41:17.762790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:17.762790+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:47.044535+0000","scrub_duration":0.107600868,"objects_trimmed":0,"snaptrim_duration":0.0022445049999999999,"stat_sum":{"num_bytes":1818624,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3637248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,0,1,5,7,12],"acting":[13,2,0,1,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.17","version":"100'80","reported_seq":309,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.086791+0000","last_change":"2024-09-15T03:42:07.087678+0000","last_active":"2024-09-15T03:42:07.086791+0000","last_peered":"2024-09-15T03:42:07.086791+0000","last_clean":"2024-09-15T03:42:07.086791+0000","last_became_active":"2024-09-15T03:41:19.031644+0000","last_became_peered":"2024-09-15T03:41:19.031644+0000","last_unstale":"2024-09-15T03:42:07.086791+0000","last_undegraded":"2024-09-15T03:42:07.086791+0000","last_fullsized":"2024-09-15T03:42:07.086791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'40","last_scrub_stamp":"2024-09-15T03:41:30.774099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.774099+0000","objects_scrubbed":3,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:49.879526+0000","scrub_duration":0.003155648,"objects_trimmed":0,"snaptrim_duration":0.012574444000000001,"stat_sum":{"num_bytes":8486912,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":723,"num_write":16,"num_write_kb":4160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,10,9,7,1,13],"acting":[5,15,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.16","version":"39'12","reported_seq":210,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027639+0000","last_change":"2024-09-15T03:42:02.027357+0000","last_active":"2024-09-15T03:42:05.027639+0000","last_peered":"2024-09-15T03:42:05.027639+0000","last_clean":"2024-09-15T03:42:05.027639+0000","last_became_active":"2024-09-15T03:41:50.016734+0000","last_became_peered":"2024-09-15T03:41:50.016734+0000","last_unstale":"2024-09-15T03:42:05.027639+0000","last_undegraded":"2024-09-15T03:42:05.027639+0000","last_fullsized":"2024-09-15T03:42:05.027639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'12","last_scrub_stamp":"2024-09-15T03:41:32.804081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:32.804081+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:40.140365+0000","scrub_duration":0.0022082680000000002,"objects_trimmed":0,"snaptrim_duration":0.0010694229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,13,8,9,5,10],"acting":[11,15,13,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.15","version":"100'46","reported_seq":235,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.083756+0000","last_change":"2024-09-15T03:42:07.084720+0000","last_active":"2024-09-15T03:42:07.083756+0000","last_peered":"2024-09-15T03:42:07.083756+0000","last_clean":"2024-09-15T03:42:07.083756+0000","last_became_active":"2024-09-15T03:41:19.032261+0000","last_became_peered":"2024-09-15T03:41:19.032261+0000","last_unstale":"2024-09-15T03:42:07.083756+0000","last_undegraded":"2024-09-15T03:42:07.083756+0000","last_fullsized":"2024-09-15T03:42:07.083756+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'17","last_scrub_stamp":"2024-09-15T03:41:28.504954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:28.504954+0000","objects_scrubbed":3,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:36.482590+0000","scrub_duration":0.0034390330000000002,"objects_trimmed":0,"snaptrim_duration":0.0095388279999999992,"stat_sum":{"num_bytes":4407296,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":11332,"num_write":12,"num_write_kb":9200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0,13,14,2,15],"acting":[9,6,0,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":204,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027696+0000","last_change":"2024-09-15T03:42:02.026719+0000","last_active":"2024-09-15T03:42:05.027696+0000","last_peered":"2024-09-15T03:42:05.027696+0000","last_clean":"2024-09-15T03:42:05.027696+0000","last_became_active":"2024-09-15T03:41:47.141772+0000","last_became_peered":"2024-09-15T03:41:47.141772+0000","last_unstale":"2024-09-15T03:42:05.027696+0000","last_undegraded":"2024-09-15T03:42:05.027696+0000","last_fullsized":"2024-09-15T03:42:05.027696+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:52.583540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:52.583540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.661100+0000","scrub_duration":0.0034973769999999999,"objects_trimmed":0,"snaptrim_duration":0.00057412800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,15,1,13,8,7],"acting":[11,9,15,1,13,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":94,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.044229+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:05.044229+0000","last_peered":"2024-09-15T03:42:05.044229+0000","last_clean":"2024-09-15T03:42:05.044229+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:05.044229+0000","last_undegraded":"2024-09-15T03:42:05.044229+0000","last_fullsized":"2024-09-15T03:42:05.044229+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"101'22","reported_seq":276,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-15T03:42:09.355769+0000","last_change":"2024-09-15T03:42:07.070405+0000","last_active":"2024-09-15T03:42:09.355769+0000","last_peered":"2024-09-15T03:42:09.355769+0000","last_clean":"2024-09-15T03:42:09.355769+0000","last_became_active":"2024-09-15T03:41:12.026344+0000","last_became_peered":"2024-09-15T03:41:12.026344+0000","last_unstale":"2024-09-15T03:42:09.355769+0000","last_undegraded":"2024-09-15T03:42:09.355769+0000","last_fullsized":"2024-09-15T03:42:09.355769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12","last_scrub_stamp":"2024-09-15T03:41:30.551305+0000","last_deep_scrub":"39'12","last_deep_scrub_stamp":"2024-09-15T03:41:30.551305+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.551305+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:57.833030+0000","scrub_duration":0.0038834,"objects_trimmed":0,"snaptrim_duration":0.00064149599999999997,"stat_sum":{"num_bytes":5652480,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":9652,"num_write":9,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.254148+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:06.254148+0000","last_peered":"2024-09-15T03:42:06.254148+0000","last_clean":"2024-09-15T03:42:06.254148+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:06.254148+0000","last_undegraded":"2024-09-15T03:42:06.254148+0000","last_fullsized":"2024-09-15T03:42:06.254148+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"94'42","reported_seq":258,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071048+0000","last_change":"2024-09-15T03:42:07.071359+0000","last_active":"2024-09-15T03:42:07.071048+0000","last_peered":"2024-09-15T03:42:07.071048+0000","last_clean":"2024-09-15T03:42:07.071048+0000","last_became_active":"2024-09-15T03:41:50.016701+0000","last_became_peered":"2024-09-15T03:41:50.016701+0000","last_unstale":"2024-09-15T03:42:07.071048+0000","last_undegraded":"2024-09-15T03:42:07.071048+0000","last_fullsized":"2024-09-15T03:42:07.071048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'23","last_scrub_stamp":"2024-09-15T03:41:27.592167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:27.592167+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:51.134452+0000","scrub_duration":0.123318016,"objects_trimmed":0,"snaptrim_duration":0.0066503029999999998,"stat_sum":{"num_bytes":1736704,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,6,9,0,8,5,11],"acting":[14,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.145764+0000","last_change":"2024-09-15T03:42:07.145738+0000","last_active":"2024-09-15T03:42:07.145764+0000","last_peered":"2024-09-15T03:42:07.145764+0000","last_clean":"2024-09-15T03:42:07.145764+0000","last_became_active":"2024-09-15T03:41:53.148724+0000","last_became_peered":"2024-09-15T03:41:53.148724+0000","last_unstale":"2024-09-15T03:42:07.145764+0000","last_undegraded":"2024-09-15T03:42:07.145764+0000","last_fullsized":"2024-09-15T03:42:07.145764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:28.348770+0000","scrub_duration":0.0017459419999999999,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":250,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.065161+0000","last_change":"2024-09-15T03:42:07.065801+0000","last_active":"2024-09-15T03:42:07.065161+0000","last_peered":"2024-09-15T03:42:07.065161+0000","last_clean":"2024-09-15T03:42:07.065161+0000","last_became_active":"2024-09-15T03:41:47.145039+0000","last_became_peered":"2024-09-15T03:41:47.145039+0000","last_unstale":"2024-09-15T03:42:07.065161+0000","last_undegraded":"2024-09-15T03:42:07.065161+0000","last_fullsized":"2024-09-15T03:42:07.065161+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:39.849670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:39.849670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:45.081635+0000","scrub_duration":0.0023338590000000002,"objects_trimmed":0,"snaptrim_duration":0.00092721299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,5,1,0,14,13],"acting":[8,2,5,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-15T03:42:09.355622+0000","last_change":"2024-09-15T03:41:47.140044+0000","last_active":"2024-09-15T03:42:09.355622+0000","last_peered":"2024-09-15T03:42:09.355622+0000","last_clean":"2024-09-15T03:42:09.355622+0000","last_became_active":"2024-09-15T03:41:47.139701+0000","last_became_peered":"2024-09-15T03:41:47.139701+0000","last_unstale":"2024-09-15T03:42:09.355622+0000","last_undegraded":"2024-09-15T03:42:09.355622+0000","last_fullsized":"2024-09-15T03:42:09.355622+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"95'56","reported_seq":335,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.013290+0000","last_change":"2024-09-15T03:42:02.029462+0000","last_active":"2024-09-15T03:42:05.013290+0000","last_peered":"2024-09-15T03:42:05.013290+0000","last_clean":"2024-09-15T03:42:05.013290+0000","last_became_active":"2024-09-15T03:41:50.021802+0000","last_became_peered":"2024-09-15T03:41:50.021802+0000","last_unstale":"2024-09-15T03:42:05.013290+0000","last_undegraded":"2024-09-15T03:42:05.013290+0000","last_fullsized":"2024-09-15T03:42:05.013290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-15T03:41:15.701370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.701370+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:33.489446+0000","scrub_duration":0.0086709649999999992,"objects_trimmed":0,"snaptrim_duration":0.003164802,"stat_sum":{"num_bytes":2424832,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":18,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,5,13,7,0,6],"acting":[12,14,5,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.3","version":"100'19","reported_seq":319,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071505+0000","last_change":"2024-09-15T03:42:07.071505+0000","last_active":"2024-09-15T03:42:07.071505+0000","last_peered":"2024-09-15T03:42:07.071505+0000","last_clean":"2024-09-15T03:42:07.071505+0000","last_became_active":"2024-09-15T03:41:50.021721+0000","last_became_peered":"2024-09-15T03:41:50.021721+0000","last_unstale":"2024-09-15T03:42:07.071505+0000","last_undegraded":"2024-09-15T03:42:07.071505+0000","last_fullsized":"2024-09-15T03:42:07.071505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'18","last_scrub_stamp":"2024-09-15T03:41:40.701317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:40.701317+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:07.556129+0000","scrub_duration":0.0022822810000000002,"objects_trimmed":0,"snaptrim_duration":0.001724777,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,9,15,2,5,6],"acting":[13,7,9,15,2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":101,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.013218+0000","last_change":"2024-09-15T03:41:55.835706+0000","last_active":"2024-09-15T03:42:05.013218+0000","last_peered":"2024-09-15T03:42:05.013218+0000","last_clean":"2024-09-15T03:42:05.013218+0000","last_became_active":"2024-09-15T03:41:19.030870+0000","last_became_peered":"2024-09-15T03:41:19.030870+0000","last_unstale":"2024-09-15T03:42:05.013218+0000","last_undegraded":"2024-09-15T03:42:05.013218+0000","last_fullsized":"2024-09-15T03:42:05.013218+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:07.326078+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,5],"acting":[12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"100'44","reported_seq":332,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.069385+0000","last_change":"2024-09-15T03:42:07.070833+0000","last_active":"2024-09-15T03:42:07.069385+0000","last_peered":"2024-09-15T03:42:07.069385+0000","last_clean":"2024-09-15T03:42:07.069385+0000","last_became_active":"2024-09-15T03:41:50.021213+0000","last_became_peered":"2024-09-15T03:41:50.021213+0000","last_unstale":"2024-09-15T03:42:07.069385+0000","last_undegraded":"2024-09-15T03:42:07.069385+0000","last_fullsized":"2024-09-15T03:42:07.069385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'14","last_scrub_stamp":"2024-09-15T03:41:15.716351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.716351+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:22.481352+0000","scrub_duration":0.0021503820000000002,"objects_trimmed":0,"snaptrim_duration":0.0063155429999999999,"stat_sum":{"num_bytes":4227072,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":16,"num_read_kb":1157,"num_write":11,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,8],"acting":[14,2,15,1,5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.083608+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:06.083608+0000","last_peered":"2024-09-15T03:42:06.083608+0000","last_clean":"2024-09-15T03:42:06.083608+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:06.083608+0000","last_undegraded":"2024-09-15T03:42:06.083608+0000","last_fullsized":"2024-09-15T03:42:06.083608+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"95'24","reported_seq":262,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.065056+0000","last_change":"2024-09-15T03:42:07.065883+0000","last_active":"2024-09-15T03:42:07.065056+0000","last_peered":"2024-09-15T03:42:07.065056+0000","last_clean":"2024-09-15T03:42:07.065056+0000","last_became_active":"2024-09-15T03:41:50.026461+0000","last_became_peered":"2024-09-15T03:41:50.026461+0000","last_unstale":"2024-09-15T03:42:07.065056+0000","last_undegraded":"2024-09-15T03:42:07.065056+0000","last_fullsized":"2024-09-15T03:42:07.065056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'9","last_scrub_stamp":"2024-09-15T03:41:20.521896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:20.521896+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:39.139808+0000","scrub_duration":0.0032032549999999999,"objects_trimmed":0,"snaptrim_duration":0.0010649539999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":721,"num_write":22,"num_write_kb":240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,9,5,10,2,13],"acting":[8,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.058387+0000","last_change":"2024-09-15T03:42:05.071989+0000","last_active":"2024-09-15T03:42:06.058387+0000","last_peered":"2024-09-15T03:42:06.058387+0000","last_clean":"2024-09-15T03:42:06.058387+0000","last_became_active":"2024-09-15T03:41:53.909931+0000","last_became_peered":"2024-09-15T03:41:53.909931+0000","last_unstale":"2024-09-15T03:42:06.058387+0000","last_undegraded":"2024-09-15T03:42:06.058387+0000","last_fullsized":"2024-09-15T03:42:06.058387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.252934+0000","scrub_duration":0.0014713300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"35'2","reported_seq":268,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.072189+0000","last_change":"2024-09-15T03:42:07.072798+0000","last_active":"2024-09-15T03:42:07.072189+0000","last_peered":"2024-09-15T03:42:07.072189+0000","last_clean":"2024-09-15T03:42:07.072189+0000","last_became_active":"2024-09-15T03:41:19.036787+0000","last_became_peered":"2024-09-15T03:41:19.036787+0000","last_unstale":"2024-09-15T03:42:07.072189+0000","last_undegraded":"2024-09-15T03:42:07.072189+0000","last_fullsized":"2024-09-15T03:42:07.072189+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-15T03:41:44.574995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:44.574995+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:48.638764+0000","scrub_duration":0.82793730399999999,"objects_trimmed":0,"snaptrim_duration":0.0027774520000000001,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,7,6,15,1,11],"acting":[13,2,7,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":86,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.059109+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:42:06.059109+0000","last_peered":"2024-09-15T03:42:06.059109+0000","last_clean":"2024-09-15T03:42:06.059109+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:42:06.059109+0000","last_undegraded":"2024-09-15T03:42:06.059109+0000","last_fullsized":"2024-09-15T03:42:06.059109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":229,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.084011+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:06.084011+0000","last_peered":"2024-09-15T03:42:06.084011+0000","last_clean":"2024-09-15T03:42:06.084011+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:06.084011+0000","last_undegraded":"2024-09-15T03:42:06.084011+0000","last_fullsized":"2024-09-15T03:42:06.084011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"100'88","reported_seq":339,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.083764+0000","last_change":"2024-09-15T03:42:07.084671+0000","last_active":"2024-09-15T03:42:07.083764+0000","last_peered":"2024-09-15T03:42:07.083764+0000","last_clean":"2024-09-15T03:42:07.083764+0000","last_became_active":"2024-09-15T03:41:12.029274+0000","last_became_peered":"2024-09-15T03:41:12.029274+0000","last_unstale":"2024-09-15T03:42:07.083764+0000","last_undegraded":"2024-09-15T03:42:07.083764+0000","last_fullsized":"2024-09-15T03:42:07.083764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13","last_scrub_stamp":"2024-09-15T03:41:15.740279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.740279+0000","objects_scrubbed":4,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:38.870215+0000","scrub_duration":0.0036761319999999999,"objects_trimmed":0,"snaptrim_duration":0.012832509000000001,"stat_sum":{"num_bytes":7061504,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":32096,"num_write":82,"num_write_kb":9936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.9","version":"35'7","reported_seq":271,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.064747+0000","last_change":"2024-09-15T03:42:07.065065+0000","last_active":"2024-09-15T03:42:07.064747+0000","last_peered":"2024-09-15T03:42:07.064747+0000","last_clean":"2024-09-15T03:42:07.064747+0000","last_became_active":"2024-09-15T03:41:19.035847+0000","last_became_peered":"2024-09-15T03:41:19.035847+0000","last_unstale":"2024-09-15T03:42:07.064747+0000","last_undegraded":"2024-09-15T03:42:07.064747+0000","last_fullsized":"2024-09-15T03:42:07.064747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-09-15T03:41:30.900331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.900331+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:41.518622+0000","scrub_duration":0.0023468389999999999,"objects_trimmed":0,"snaptrim_duration":0.00050368400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,12,15,8,10],"acting":[14,1,0,12,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.a","version":"100'86","reported_seq":308,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.075710+0000","last_change":"2024-09-15T03:42:07.075710+0000","last_active":"2024-09-15T03:42:07.075710+0000","last_peered":"2024-09-15T03:42:07.075710+0000","last_clean":"2024-09-15T03:42:07.075710+0000","last_became_active":"2024-09-15T03:41:19.034361+0000","last_became_peered":"2024-09-15T03:41:19.034361+0000","last_unstale":"2024-09-15T03:42:07.075710+0000","last_undegraded":"2024-09-15T03:42:07.075710+0000","last_fullsized":"2024-09-15T03:42:07.075710+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:11.092582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010695812000000001,"stat_sum":{"num_bytes":9781248,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":10,"num_write":45,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,1,9,14,6,11],"acting":[8,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.b","version":"100'42","reported_seq":345,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.081270+0000","last_change":"2024-09-15T03:42:07.083548+0000","last_active":"2024-09-15T03:42:07.081270+0000","last_peered":"2024-09-15T03:42:07.081270+0000","last_clean":"2024-09-15T03:42:07.081270+0000","last_became_active":"2024-09-15T03:41:50.018104+0000","last_became_peered":"2024-09-15T03:41:50.018104+0000","last_unstale":"2024-09-15T03:42:07.081270+0000","last_undegraded":"2024-09-15T03:42:07.081270+0000","last_fullsized":"2024-09-15T03:42:07.081270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'40","last_scrub_stamp":"2024-09-15T03:42:06.615211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:42:06.615211+0000","objects_scrubbed":4,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.541627+0000","scrub_duration":0.0033695489999999999,"objects_trimmed":0,"snaptrim_duration":0.018796686,"stat_sum":{"num_bytes":4653056,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":12296,"num_write":40,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5,1,12,6,7,9],"acting":[14,5,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.c","version":"100'57","reported_seq":298,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071107+0000","last_change":"2024-09-15T03:42:07.071701+0000","last_active":"2024-09-15T03:42:07.071107+0000","last_peered":"2024-09-15T03:42:07.071107+0000","last_clean":"2024-09-15T03:42:07.071107+0000","last_became_active":"2024-09-15T03:41:39.015749+0000","last_became_peered":"2024-09-15T03:41:39.015749+0000","last_unstale":"2024-09-15T03:42:07.071107+0000","last_undegraded":"2024-09-15T03:42:07.071107+0000","last_fullsized":"2024-09-15T03:42:07.071107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:11.128571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070989579999999998,"stat_sum":{"num_bytes":2244608,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14508,"num_write":47,"num_write_kb":7008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,0,10,6,12,5],"acting":[14,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.d","version":"94'42","reported_seq":329,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.019201+0000","last_change":"2024-09-15T03:42:02.016295+0000","last_active":"2024-09-15T03:42:05.019201+0000","last_peered":"2024-09-15T03:42:05.019201+0000","last_clean":"2024-09-15T03:42:05.019201+0000","last_became_active":"2024-09-15T03:41:50.021249+0000","last_became_peered":"2024-09-15T03:41:50.021249+0000","last_unstale":"2024-09-15T03:42:05.019201+0000","last_undegraded":"2024-09-15T03:42:05.019201+0000","last_fullsized":"2024-09-15T03:42:05.019201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038017699999999999,"stat_sum":{"num_bytes":1982464,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10422,"num_write":24,"num_write_kb":7200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,14,10,9,8,2],"acting":[7,0,14,10,9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.e","version":"95'87","reported_seq":377,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.360189+0000","last_change":"2024-09-15T03:42:07.360129+0000","last_active":"2024-09-15T03:42:07.360189+0000","last_peered":"2024-09-15T03:42:07.360189+0000","last_clean":"2024-09-15T03:42:07.360189+0000","last_became_active":"2024-09-15T03:41:39.015840+0000","last_became_peered":"2024-09-15T03:41:39.015840+0000","last_unstale":"2024-09-15T03:42:07.360189+0000","last_undegraded":"2024-09-15T03:42:07.360189+0000","last_fullsized":"2024-09-15T03:42:07.360189+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'87","last_scrub_stamp":"2024-09-15T03:42:07.360044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.360044+0000","objects_scrubbed":3,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:11.109168+0000","scrub_duration":0.0027624279999999999,"objects_trimmed":0,"snaptrim_duration":0.00056164799999999999,"stat_sum":{"num_bytes":5685248,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":25407,"num_write":29,"num_write_kb":11328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,1,11,8,6,12],"acting":[2,10,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.f","version":"97'89","reported_seq":291,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027726+0000","last_change":"2024-09-15T03:42:02.027714+0000","last_active":"2024-09-15T03:42:05.027726+0000","last_peered":"2024-09-15T03:42:05.027726+0000","last_clean":"2024-09-15T03:42:05.027726+0000","last_became_active":"2024-09-15T03:40:59.584644+0000","last_became_peered":"2024-09-15T03:40:59.584644+0000","last_unstale":"2024-09-15T03:42:05.027726+0000","last_undegraded":"2024-09-15T03:42:05.027726+0000","last_fullsized":"2024-09-15T03:42:05.027726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:07.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001140117,"stat_sum":{"num_bytes":12206080,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":1978,"num_write":98,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.10","version":"95'69","reported_seq":290,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.070885+0000","last_change":"2024-09-15T03:42:07.074650+0000","last_active":"2024-09-15T03:42:07.070885+0000","last_peered":"2024-09-15T03:42:07.070885+0000","last_clean":"2024-09-15T03:42:07.070885+0000","last_became_active":"2024-09-15T03:41:50.019077+0000","last_became_peered":"2024-09-15T03:41:50.019077+0000","last_unstale":"2024-09-15T03:42:07.070885+0000","last_undegraded":"2024-09-15T03:42:07.070885+0000","last_fullsized":"2024-09-15T03:42:07.070885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'33","last_scrub_stamp":"2024-09-15T03:41:29.106893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:29.106893+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:49.054548+0000","scrub_duration":0.39459468800000003,"objects_trimmed":0,"snaptrim_duration":0.0048221890000000002,"stat_sum":{"num_bytes":4423680,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":579,"num_write":12,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,5,8,0,11],"acting":[13,6,7,5,8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.11","version":"71'31","reported_seq":275,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.069591+0000","last_change":"2024-09-15T03:42:07.070028+0000","last_active":"2024-09-15T03:42:07.069591+0000","last_peered":"2024-09-15T03:42:07.069591+0000","last_clean":"2024-09-15T03:42:07.069591+0000","last_became_active":"2024-09-15T03:41:17.030558+0000","last_became_peered":"2024-09-15T03:41:17.030558+0000","last_unstale":"2024-09-15T03:42:07.069591+0000","last_undegraded":"2024-09-15T03:42:07.069591+0000","last_fullsized":"2024-09-15T03:42:07.069591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'17","last_scrub_stamp":"2024-09-15T03:41:17.535319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:17.535319+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:20.364025+0000","scrub_duration":0.003091599,"objects_trimmed":0,"snaptrim_duration":0.00056524600000000004,"stat_sum":{"num_bytes":3309568,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":5188,"num_write":7,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.12","version":"35'2","reported_seq":218,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.074859+0000","last_change":"2024-09-15T03:42:07.075474+0000","last_active":"2024-09-15T03:42:07.074859+0000","last_peered":"2024-09-15T03:42:07.074859+0000","last_clean":"2024-09-15T03:42:07.074859+0000","last_became_active":"2024-09-15T03:41:50.026297+0000","last_became_peered":"2024-09-15T03:41:50.026297+0000","last_unstale":"2024-09-15T03:42:07.074859+0000","last_undegraded":"2024-09-15T03:42:07.074859+0000","last_fullsized":"2024-09-15T03:42:07.074859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-15T03:41:29.466491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:29.466491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:36.023400+0000","scrub_duration":0.0021447580000000001,"objects_trimmed":0,"snaptrim_duration":0.000861165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,11,2,15],"acting":[9,7,8,6,11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.13","version":"94'64","reported_seq":385,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027659+0000","last_change":"2024-09-15T03:42:02.027314+0000","last_active":"2024-09-15T03:42:05.027659+0000","last_peered":"2024-09-15T03:42:05.027659+0000","last_clean":"2024-09-15T03:42:05.027659+0000","last_became_active":"2024-09-15T03:41:50.024851+0000","last_became_peered":"2024-09-15T03:41:50.024851+0000","last_unstale":"2024-09-15T03:42:05.027659+0000","last_undegraded":"2024-09-15T03:42:05.027659+0000","last_fullsized":"2024-09-15T03:42:05.027659+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'21","last_scrub_stamp":"2024-09-15T03:41:19.809341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:19.809341+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:21.113176+0000","scrub_duration":0.0033873140000000002,"objects_trimmed":0,"snaptrim_duration":0.001305797,"stat_sum":{"num_bytes":3227648,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":18,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,5,2,0,15,10],"acting":[11,7,5,2,0,15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":88539136,"num_objects":54,"num_object_clones":13,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":3,"num_read":279,"num_read_kb":144702,"num_write":503,"num_write_kb":109408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":53411840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159145984,"data_stored":189804544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":0},"log_size":1056,"ondisk_log_size":1056,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280531,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313528,"kb_used_data":12008,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058312,"statfs":{"total":96636764160,"available":96315711488,"internally_reserved":0,"allocated":12296192,"data_stored":13461859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":170.74299999999999,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6849999999999996,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.501},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6980000000000004,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.85899999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.75900000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.77400000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":23.472999999999999,"5min":23.472999999999999,"15min":23.472999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":478.76900000000001,"5min":478.76900000000001,"15min":478.76900000000001},"last":0.996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.80000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":11,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.68100000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":1.012},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.878}]}]},{"osd":14,"up_from":22,"seq":94489280532,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315348,"kb_used_data":13828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056492,"statfs":{"total":96636764160,"available":96313847808,"internally_reserved":0,"allocated":14159872,"data_stored":15865667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.60199999999999998},{"interface":"front","average":{"1min":48.625999999999998,"5min":48.625999999999998,"15min":48.625999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":913.03200000000004,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.79100000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.629},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91600000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.754}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":30.189,"5min":30.189,"15min":30.189},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":560.85299999999995,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.85899999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":1.03},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":1.141}]}]},{"osd":13,"up_from":22,"seq":94489280531,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313776,"kb_used_data":12260,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058064,"statfs":{"total":96636764160,"available":96315457536,"internally_reserved":0,"allocated":12554240,"data_stored":14011280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.90400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":4,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.876}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.83099999999999996},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.81299999999999994}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.145}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":10,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":18.027999999999999,"5min":18.027999999999999,"15min":18.027999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":379.39100000000002,"5min":379.39100000000002,"15min":379.39100000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":41.061999999999998,"5min":41.061999999999998,"15min":41.061999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":886.51800000000003,"5min":886.51800000000003,"15min":886.51800000000003},"last":1.2090000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":1.1850000000000001},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":1.0309999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.288}]},{"osd":14,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":13.311,"5min":13.311,"15min":13.311},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":274.65100000000001,"5min":274.65100000000001,"15min":274.65100000000001},"last":1.2450000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":1.321}]}]},{"osd":0,"up_from":21,"seq":90194313237,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315124,"kb_used_data":13604,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056716,"statfs":{"total":96636764160,"available":96314077184,"internally_reserved":0,"allocated":13930496,"data_stored":18755720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":41.966000000000001,"5min":41.966000000000001,"15min":41.966000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":703.98400000000004,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.745}]},{"osd":7,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":25.172999999999998,"5min":25.172999999999998,"15min":25.172999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":419.01499999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.68200000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.39800000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70199999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":3.972,"5min":3.972,"15min":3.972},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":62.659999999999997,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.80100000000000005}]}]},{"osd":6,"up_from":21,"seq":90194313235,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315268,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056572,"statfs":{"total":96636764160,"available":96313929728,"internally_reserved":0,"allocated":14082048,"data_stored":15658520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":37.668999999999997,"5min":37.668999999999997,"15min":37.668999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":668.923,"5min":668.923,"15min":668.923},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51700000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":15.977,"5min":15.977,"15min":15.977},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":290.32400000000001,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.60099999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.58899999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70299999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.77200000000000002}]}]},{"osd":15,"up_from":21,"seq":90194313237,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306640,"kb_used_data":9276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065200,"statfs":{"total":96636764160,"available":96322764800,"internally_reserved":0,"allocated":9498624,"data_stored":11354867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":45.473999999999997,"5min":45.473999999999997,"15min":45.473999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":764.62199999999996,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.63100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68700000000000006}]},{"osd":5,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.442}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":18.594999999999999,"5min":18.594999999999999,"15min":18.594999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":305.18200000000002,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.54900000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":58.883000000000003,"5min":58.883000000000003,"15min":58.883000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":782.05200000000002,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47699999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.67700000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.69599999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58099999999999996}]},{"osd":14,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65500000000000003}]}]},{"osd":3,"up_from":0,"seq":292057776137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":22,"seq":94489280531,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313432,"kb_used_data":11908,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058408,"statfs":{"total":96636764160,"available":96315809792,"internally_reserved":0,"allocated":12193792,"data_stored":12408117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.375},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.64200000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.58199999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.71299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":40.006999999999998,"5min":40.006999999999998,"15min":40.006999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":740.69799999999998,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.70299999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.66600000000000004}]}]},{"osd":11,"up_from":22,"seq":94489280531,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314072,"kb_used_data":12556,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94057768,"statfs":{"total":96636764160,"available":96315154432,"internally_reserved":0,"allocated":12857344,"data_stored":12394008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.58099999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":892.99000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.64000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.177,"5min":3.177,"15min":3.177},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.71599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":33.399999999999999,"5min":33.399999999999999,"15min":33.399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":557.40099999999995,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.77600000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.93100000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":41.654000000000003,"5min":41.654000000000003,"15min":41.654000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":694.31399999999996,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.91900000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":11.574,"5min":11.574,"15min":11.574},"last":0.81899999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.96099999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":12,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304840,"kb_used_data":7484,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94067000,"statfs":{"total":96636764160,"available":96324608000,"internally_reserved":0,"allocated":7663616,"data_stored":7601129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.318},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.43099999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":10.916,"5min":10.916,"15min":10.916},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":201.73599999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.54700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":14.640000000000001,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.39700000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":818.68700000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.56399999999999995}]},{"osd":15,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":34.192,"5min":34.192,"15min":34.192},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":703.52499999999998,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.55500000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280531,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311848,"kb_used_data":14492,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94059992,"statfs":{"total":96636764160,"available":96317431808,"internally_reserved":0,"allocated":14839808,"data_stored":16931798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.44},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.34399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":10.869999999999999,"5min":10.869999999999999,"15min":10.869999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":214.91,"5min":214.91,"15min":214.91},"last":0.45600000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.624}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46899999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.48899999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.61199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.010000000000002}]}]},{"osd":4,"up_from":0,"seq":94489280532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":22,"seq":94489280531,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313612,"kb_used_data":12092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058228,"statfs":{"total":96636764160,"available":96315625472,"internally_reserved":0,"allocated":12382208,"data_stored":15000852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.34899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.438},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.48999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.47499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.53100000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.71099999999999997}]},{"osd":15,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.84299999999999997}]}]},{"osd":8,"up_from":22,"seq":94489280532,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":12912,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":13221888,"data_stored":13289283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.377}]},{"osd":1,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.503}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":903.59500000000003}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.57799999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":22.788,"5min":22.788,"15min":22.788},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":398.71699999999998,"5min":398.71699999999998,"15min":398.71699999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55600000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":51.093000000000004,"5min":51.093000000000004,"15min":51.093000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":908.44200000000001,"5min":908.44200000000001,"15min":908.44200000000001},"last":0.53200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":8.952,"5min":8.952,"15min":8.952},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":149.09,"5min":149.09,"15min":149.09},"last":0.57099999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.62},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.629}]},{"osd":14,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.68999999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313300,"kb_used_data":11780,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058540,"statfs":{"total":96636764160,"available":96315944960,"internally_reserved":0,"allocated":12062720,"data_stored":13731651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":10.802,"5min":10.802,"15min":10.802},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":151.80600000000001,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.70599999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":7.327,"5min":7.327,"15min":7.327},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":98.858000000000004,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.88400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":37.173999999999999,"5min":37.173999999999999,"15min":37.173999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":14.319000000000001,"5min":14.319000000000001,"15min":14.319000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":203.91399999999999,"5min":203.91399999999999,"15min":203.91399999999999},"last":0.82099999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.94699999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.85399999999999998},{"interface":"front","average":{"1min":5.7759999999999998,"5min":5.7759999999999998,"15min":5.7759999999999998},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":73.846999999999994,"5min":73.846999999999994,"15min":73.846999999999994},"last":0.871}]},{"osd":15,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.80600000000000005}]}]},{"osd":10,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":10512,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":10764288,"data_stored":12314435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57299999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.65700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69899999999999995},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.70999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":16.597000000000001,"5min":16.597000000000001,"15min":16.597000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":287.86900000000003,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12406784,"data_stored":17850368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13381632,"data_stored":16068608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":12058624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6684672,"data_stored":7659520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10903552,"data_stored":14127104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":14794752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11325440,"data_stored":12808192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11743232,"data_stored":12414976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10584064,"data_stored":12500992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9285632,"data_stored":11440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11395072,"data_stored":11530240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6201344,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":10567680,"data_stored":12619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12681216,"data_stored":14991360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8634368,"data_stored":11755520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:11.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:11.505 INFO:tasks.rados.rados.0.smithi105.stdout:378: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-15T03:42:11.592 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:11.591+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:11.599 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:42:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-15T03:42:11.599 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:11.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-15T03:42:11.693 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:11.693+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:11.750 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:11.750 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:42:11.750 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:42:11.750 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:42:11.751 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:11.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-15T03:42:11.794 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:11.794+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:11.825 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:11.826 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:11.836 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-15T03:42:11.016048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89129523,"num_objects":58,"num_object_clones":13,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":3,"num_read":365,"num_read_kb":144775,"num_write":650,"num_write_kb":112082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":53411859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1171,"ondisk_log_size":1171,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":4373092,"kb_used_data":168464,"kb_used_omap":23,"kb_used_meta":4204392,"kb_avail":1316832668,"statfs":{"total":1352914698240,"available":1348436652032,"internally_reserved":0,"allocated":172507136,"data_stored":192779186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23856,"internal_metadata":4305298128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11878400,"num_objects":13,"num_object_clones":12,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":21,"num_read_kb":10424,"num_write":91,"num_write_kb":11600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.953528"},"pg_stats":[{"pgid":"3.19","version":"70'27","reported_seq":297,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.066089+0000","last_change":"2024-09-15T03:42:07.066744+0000","last_active":"2024-09-15T03:42:07.066089+0000","last_peered":"2024-09-15T03:42:07.066089+0000","last_clean":"2024-09-15T03:42:07.066089+0000","last_became_active":"2024-09-15T03:41:50.026750+0000","last_became_peered":"2024-09-15T03:41:50.026750+0000","last_unstale":"2024-09-15T03:42:07.066089+0000","last_undegraded":"2024-09-15T03:42:07.066089+0000","last_fullsized":"2024-09-15T03:42:07.066089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'21","last_scrub_stamp":"2024-09-15T03:41:30.737422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.737422+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:59.283558+0000","scrub_duration":0.0027345220000000001,"objects_trimmed":0,"snaptrim_duration":0.001624871,"stat_sum":{"num_bytes":4685824,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":18626,"num_write":2,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12,11,9,13,7,1],"acting":[8,12,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.18","version":"58'18","reported_seq":263,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071574+0000","last_change":"2024-09-15T03:42:07.072210+0000","last_active":"2024-09-15T03:42:07.071574+0000","last_peered":"2024-09-15T03:42:07.071574+0000","last_clean":"2024-09-15T03:42:07.071574+0000","last_became_active":"2024-09-15T03:41:50.021784+0000","last_became_peered":"2024-09-15T03:41:50.021784+0000","last_unstale":"2024-09-15T03:42:07.071574+0000","last_undegraded":"2024-09-15T03:42:07.071574+0000","last_fullsized":"2024-09-15T03:42:07.071574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"43'9","last_scrub_stamp":"2024-09-15T03:41:17.762790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:17.762790+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:47.044535+0000","scrub_duration":0.107600868,"objects_trimmed":0,"snaptrim_duration":0.0022445049999999999,"stat_sum":{"num_bytes":1818624,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3637248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,0,1,5,7,12],"acting":[13,2,0,1,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.17","version":"100'80","reported_seq":309,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.086791+0000","last_change":"2024-09-15T03:42:07.087678+0000","last_active":"2024-09-15T03:42:07.086791+0000","last_peered":"2024-09-15T03:42:07.086791+0000","last_clean":"2024-09-15T03:42:07.086791+0000","last_became_active":"2024-09-15T03:41:19.031644+0000","last_became_peered":"2024-09-15T03:41:19.031644+0000","last_unstale":"2024-09-15T03:42:07.086791+0000","last_undegraded":"2024-09-15T03:42:07.086791+0000","last_fullsized":"2024-09-15T03:42:07.086791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'40","last_scrub_stamp":"2024-09-15T03:41:30.774099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.774099+0000","objects_scrubbed":3,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:49.879526+0000","scrub_duration":0.003155648,"objects_trimmed":0,"snaptrim_duration":0.012574444000000001,"stat_sum":{"num_bytes":8486912,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":723,"num_write":16,"num_write_kb":4160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,10,9,7,1,13],"acting":[5,15,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.16","version":"39'12","reported_seq":210,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027639+0000","last_change":"2024-09-15T03:42:02.027357+0000","last_active":"2024-09-15T03:42:05.027639+0000","last_peered":"2024-09-15T03:42:05.027639+0000","last_clean":"2024-09-15T03:42:05.027639+0000","last_became_active":"2024-09-15T03:41:50.016734+0000","last_became_peered":"2024-09-15T03:41:50.016734+0000","last_unstale":"2024-09-15T03:42:05.027639+0000","last_undegraded":"2024-09-15T03:42:05.027639+0000","last_fullsized":"2024-09-15T03:42:05.027639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'12","last_scrub_stamp":"2024-09-15T03:41:32.804081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:32.804081+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:40.140365+0000","scrub_duration":0.0022082680000000002,"objects_trimmed":0,"snaptrim_duration":0.0010694229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,13,8,9,5,10],"acting":[11,15,13,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.15","version":"100'46","reported_seq":235,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.083756+0000","last_change":"2024-09-15T03:42:07.084720+0000","last_active":"2024-09-15T03:42:07.083756+0000","last_peered":"2024-09-15T03:42:07.083756+0000","last_clean":"2024-09-15T03:42:07.083756+0000","last_became_active":"2024-09-15T03:41:19.032261+0000","last_became_peered":"2024-09-15T03:41:19.032261+0000","last_unstale":"2024-09-15T03:42:07.083756+0000","last_undegraded":"2024-09-15T03:42:07.083756+0000","last_fullsized":"2024-09-15T03:42:07.083756+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'17","last_scrub_stamp":"2024-09-15T03:41:28.504954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:28.504954+0000","objects_scrubbed":3,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:36.482590+0000","scrub_duration":0.0034390330000000002,"objects_trimmed":0,"snaptrim_duration":0.0095388279999999992,"stat_sum":{"num_bytes":4407296,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":11332,"num_write":12,"num_write_kb":9200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0,13,14,2,15],"acting":[9,6,0,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":204,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027696+0000","last_change":"2024-09-15T03:42:02.026719+0000","last_active":"2024-09-15T03:42:05.027696+0000","last_peered":"2024-09-15T03:42:05.027696+0000","last_clean":"2024-09-15T03:42:05.027696+0000","last_became_active":"2024-09-15T03:41:47.141772+0000","last_became_peered":"2024-09-15T03:41:47.141772+0000","last_unstale":"2024-09-15T03:42:05.027696+0000","last_undegraded":"2024-09-15T03:42:05.027696+0000","last_fullsized":"2024-09-15T03:42:05.027696+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:52.583540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:52.583540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.661100+0000","scrub_duration":0.0034973769999999999,"objects_trimmed":0,"snaptrim_duration":0.00057412800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,15,1,13,8,7],"acting":[11,9,15,1,13,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":94,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.044229+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:05.044229+0000","last_peered":"2024-09-15T03:42:05.044229+0000","last_clean":"2024-09-15T03:42:05.044229+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:05.044229+0000","last_undegraded":"2024-09-15T03:42:05.044229+0000","last_fullsized":"2024-09-15T03:42:05.044229+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"101'22","reported_seq":276,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-15T03:42:09.355769+0000","last_change":"2024-09-15T03:42:07.070405+0000","last_active":"2024-09-15T03:42:09.355769+0000","last_peered":"2024-09-15T03:42:09.355769+0000","last_clean":"2024-09-15T03:42:09.355769+0000","last_became_active":"2024-09-15T03:41:12.026344+0000","last_became_peered":"2024-09-15T03:41:12.026344+0000","last_unstale":"2024-09-15T03:42:09.355769+0000","last_undegraded":"2024-09-15T03:42:09.355769+0000","last_fullsized":"2024-09-15T03:42:09.355769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12","last_scrub_stamp":"2024-09-15T03:41:30.551305+0000","last_deep_scrub":"39'12","last_deep_scrub_stamp":"2024-09-15T03:41:30.551305+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.551305+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:57.833030+0000","scrub_duration":0.0038834,"objects_trimmed":0,"snaptrim_duration":0.00064149599999999997,"stat_sum":{"num_bytes":5652480,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":9652,"num_write":9,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.254148+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:06.254148+0000","last_peered":"2024-09-15T03:42:06.254148+0000","last_clean":"2024-09-15T03:42:06.254148+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:06.254148+0000","last_undegraded":"2024-09-15T03:42:06.254148+0000","last_fullsized":"2024-09-15T03:42:06.254148+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"94'42","reported_seq":258,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071048+0000","last_change":"2024-09-15T03:42:07.071359+0000","last_active":"2024-09-15T03:42:07.071048+0000","last_peered":"2024-09-15T03:42:07.071048+0000","last_clean":"2024-09-15T03:42:07.071048+0000","last_became_active":"2024-09-15T03:41:50.016701+0000","last_became_peered":"2024-09-15T03:41:50.016701+0000","last_unstale":"2024-09-15T03:42:07.071048+0000","last_undegraded":"2024-09-15T03:42:07.071048+0000","last_fullsized":"2024-09-15T03:42:07.071048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'23","last_scrub_stamp":"2024-09-15T03:41:27.592167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:27.592167+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:51.134452+0000","scrub_duration":0.123318016,"objects_trimmed":0,"snaptrim_duration":0.0066503029999999998,"stat_sum":{"num_bytes":1736704,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,6,9,0,8,5,11],"acting":[14,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.145764+0000","last_change":"2024-09-15T03:42:07.145738+0000","last_active":"2024-09-15T03:42:07.145764+0000","last_peered":"2024-09-15T03:42:07.145764+0000","last_clean":"2024-09-15T03:42:07.145764+0000","last_became_active":"2024-09-15T03:41:53.148724+0000","last_became_peered":"2024-09-15T03:41:53.148724+0000","last_unstale":"2024-09-15T03:42:07.145764+0000","last_undegraded":"2024-09-15T03:42:07.145764+0000","last_fullsized":"2024-09-15T03:42:07.145764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:28.348770+0000","scrub_duration":0.0017459419999999999,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":250,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.065161+0000","last_change":"2024-09-15T03:42:07.065801+0000","last_active":"2024-09-15T03:42:07.065161+0000","last_peered":"2024-09-15T03:42:07.065161+0000","last_clean":"2024-09-15T03:42:07.065161+0000","last_became_active":"2024-09-15T03:41:47.145039+0000","last_became_peered":"2024-09-15T03:41:47.145039+0000","last_unstale":"2024-09-15T03:42:07.065161+0000","last_undegraded":"2024-09-15T03:42:07.065161+0000","last_fullsized":"2024-09-15T03:42:07.065161+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:39.849670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:39.849670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:45.081635+0000","scrub_duration":0.0023338590000000002,"objects_trimmed":0,"snaptrim_duration":0.00092721299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,5,1,0,14,13],"acting":[8,2,5,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-15T03:42:09.355622+0000","last_change":"2024-09-15T03:41:47.140044+0000","last_active":"2024-09-15T03:42:09.355622+0000","last_peered":"2024-09-15T03:42:09.355622+0000","last_clean":"2024-09-15T03:42:09.355622+0000","last_became_active":"2024-09-15T03:41:47.139701+0000","last_became_peered":"2024-09-15T03:41:47.139701+0000","last_unstale":"2024-09-15T03:42:09.355622+0000","last_undegraded":"2024-09-15T03:42:09.355622+0000","last_fullsized":"2024-09-15T03:42:09.355622+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"95'56","reported_seq":335,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.013290+0000","last_change":"2024-09-15T03:42:02.029462+0000","last_active":"2024-09-15T03:42:05.013290+0000","last_peered":"2024-09-15T03:42:05.013290+0000","last_clean":"2024-09-15T03:42:05.013290+0000","last_became_active":"2024-09-15T03:41:50.021802+0000","last_became_peered":"2024-09-15T03:41:50.021802+0000","last_unstale":"2024-09-15T03:42:05.013290+0000","last_undegraded":"2024-09-15T03:42:05.013290+0000","last_fullsized":"2024-09-15T03:42:05.013290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-15T03:41:15.701370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.701370+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:33.489446+0000","scrub_duration":0.0086709649999999992,"objects_trimmed":0,"snaptrim_duration":0.003164802,"stat_sum":{"num_bytes":2424832,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":18,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,5,13,7,0,6],"acting":[12,14,5,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.3","version":"100'19","reported_seq":319,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071505+0000","last_change":"2024-09-15T03:42:07.071505+0000","last_active":"2024-09-15T03:42:07.071505+0000","last_peered":"2024-09-15T03:42:07.071505+0000","last_clean":"2024-09-15T03:42:07.071505+0000","last_became_active":"2024-09-15T03:41:50.021721+0000","last_became_peered":"2024-09-15T03:41:50.021721+0000","last_unstale":"2024-09-15T03:42:07.071505+0000","last_undegraded":"2024-09-15T03:42:07.071505+0000","last_fullsized":"2024-09-15T03:42:07.071505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'18","last_scrub_stamp":"2024-09-15T03:41:40.701317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:40.701317+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:07.556129+0000","scrub_duration":0.0022822810000000002,"objects_trimmed":0,"snaptrim_duration":0.001724777,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,9,15,2,5,6],"acting":[13,7,9,15,2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":101,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.013218+0000","last_change":"2024-09-15T03:41:55.835706+0000","last_active":"2024-09-15T03:42:05.013218+0000","last_peered":"2024-09-15T03:42:05.013218+0000","last_clean":"2024-09-15T03:42:05.013218+0000","last_became_active":"2024-09-15T03:41:19.030870+0000","last_became_peered":"2024-09-15T03:41:19.030870+0000","last_unstale":"2024-09-15T03:42:05.013218+0000","last_undegraded":"2024-09-15T03:42:05.013218+0000","last_fullsized":"2024-09-15T03:42:05.013218+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:07.326078+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,5],"acting":[12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"100'44","reported_seq":332,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.069385+0000","last_change":"2024-09-15T03:42:07.070833+0000","last_active":"2024-09-15T03:42:07.069385+0000","last_peered":"2024-09-15T03:42:07.069385+0000","last_clean":"2024-09-15T03:42:07.069385+0000","last_became_active":"2024-09-15T03:41:50.021213+0000","last_became_peered":"2024-09-15T03:41:50.021213+0000","last_unstale":"2024-09-15T03:42:07.069385+0000","last_undegraded":"2024-09-15T03:42:07.069385+0000","last_fullsized":"2024-09-15T03:42:07.069385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'14","last_scrub_stamp":"2024-09-15T03:41:15.716351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.716351+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:22.481352+0000","scrub_duration":0.0021503820000000002,"objects_trimmed":0,"snaptrim_duration":0.0063155429999999999,"stat_sum":{"num_bytes":4227072,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":16,"num_read_kb":1157,"num_write":11,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,8],"acting":[14,2,15,1,5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.083608+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:06.083608+0000","last_peered":"2024-09-15T03:42:06.083608+0000","last_clean":"2024-09-15T03:42:06.083608+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:06.083608+0000","last_undegraded":"2024-09-15T03:42:06.083608+0000","last_fullsized":"2024-09-15T03:42:06.083608+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"95'24","reported_seq":262,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.065056+0000","last_change":"2024-09-15T03:42:07.065883+0000","last_active":"2024-09-15T03:42:07.065056+0000","last_peered":"2024-09-15T03:42:07.065056+0000","last_clean":"2024-09-15T03:42:07.065056+0000","last_became_active":"2024-09-15T03:41:50.026461+0000","last_became_peered":"2024-09-15T03:41:50.026461+0000","last_unstale":"2024-09-15T03:42:07.065056+0000","last_undegraded":"2024-09-15T03:42:07.065056+0000","last_fullsized":"2024-09-15T03:42:07.065056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'9","last_scrub_stamp":"2024-09-15T03:41:20.521896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:20.521896+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:39.139808+0000","scrub_duration":0.0032032549999999999,"objects_trimmed":0,"snaptrim_duration":0.0010649539999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":721,"num_write":22,"num_write_kb":240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,9,5,10,2,13],"acting":[8,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.058387+0000","last_change":"2024-09-15T03:42:05.071989+0000","last_active":"2024-09-15T03:42:06.058387+0000","last_peered":"2024-09-15T03:42:06.058387+0000","last_clean":"2024-09-15T03:42:06.058387+0000","last_became_active":"2024-09-15T03:41:53.909931+0000","last_became_peered":"2024-09-15T03:41:53.909931+0000","last_unstale":"2024-09-15T03:42:06.058387+0000","last_undegraded":"2024-09-15T03:42:06.058387+0000","last_fullsized":"2024-09-15T03:42:06.058387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.252934+0000","scrub_duration":0.0014713300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"35'2","reported_seq":268,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.072189+0000","last_change":"2024-09-15T03:42:07.072798+0000","last_active":"2024-09-15T03:42:07.072189+0000","last_peered":"2024-09-15T03:42:07.072189+0000","last_clean":"2024-09-15T03:42:07.072189+0000","last_became_active":"2024-09-15T03:41:19.036787+0000","last_became_peered":"2024-09-15T03:41:19.036787+0000","last_unstale":"2024-09-15T03:42:07.072189+0000","last_undegraded":"2024-09-15T03:42:07.072189+0000","last_fullsized":"2024-09-15T03:42:07.072189+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-15T03:41:44.574995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:44.574995+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:48.638764+0000","scrub_duration":0.82793730399999999,"objects_trimmed":0,"snaptrim_duration":0.0027774520000000001,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,7,6,15,1,11],"acting":[13,2,7,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":86,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.059109+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:42:06.059109+0000","last_peered":"2024-09-15T03:42:06.059109+0000","last_clean":"2024-09-15T03:42:06.059109+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:42:06.059109+0000","last_undegraded":"2024-09-15T03:42:06.059109+0000","last_fullsized":"2024-09-15T03:42:06.059109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":229,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-15T03:42:06.084011+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:06.084011+0000","last_peered":"2024-09-15T03:42:06.084011+0000","last_clean":"2024-09-15T03:42:06.084011+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:06.084011+0000","last_undegraded":"2024-09-15T03:42:06.084011+0000","last_fullsized":"2024-09-15T03:42:06.084011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"100'88","reported_seq":339,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.083764+0000","last_change":"2024-09-15T03:42:07.084671+0000","last_active":"2024-09-15T03:42:07.083764+0000","last_peered":"2024-09-15T03:42:07.083764+0000","last_clean":"2024-09-15T03:42:07.083764+0000","last_became_active":"2024-09-15T03:41:12.029274+0000","last_became_peered":"2024-09-15T03:41:12.029274+0000","last_unstale":"2024-09-15T03:42:07.083764+0000","last_undegraded":"2024-09-15T03:42:07.083764+0000","last_fullsized":"2024-09-15T03:42:07.083764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13","last_scrub_stamp":"2024-09-15T03:41:15.740279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.740279+0000","objects_scrubbed":4,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:38.870215+0000","scrub_duration":0.0036761319999999999,"objects_trimmed":0,"snaptrim_duration":0.012832509000000001,"stat_sum":{"num_bytes":7061504,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":32096,"num_write":82,"num_write_kb":9936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.9","version":"35'7","reported_seq":271,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.064747+0000","last_change":"2024-09-15T03:42:07.065065+0000","last_active":"2024-09-15T03:42:07.064747+0000","last_peered":"2024-09-15T03:42:07.064747+0000","last_clean":"2024-09-15T03:42:07.064747+0000","last_became_active":"2024-09-15T03:41:19.035847+0000","last_became_peered":"2024-09-15T03:41:19.035847+0000","last_unstale":"2024-09-15T03:42:07.064747+0000","last_undegraded":"2024-09-15T03:42:07.064747+0000","last_fullsized":"2024-09-15T03:42:07.064747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-09-15T03:41:30.900331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.900331+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:41.518622+0000","scrub_duration":0.0023468389999999999,"objects_trimmed":0,"snaptrim_duration":0.00050368400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,12,15,8,10],"acting":[14,1,0,12,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.a","version":"100'86","reported_seq":308,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.075710+0000","last_change":"2024-09-15T03:42:07.075710+0000","last_active":"2024-09-15T03:42:07.075710+0000","last_peered":"2024-09-15T03:42:07.075710+0000","last_clean":"2024-09-15T03:42:07.075710+0000","last_became_active":"2024-09-15T03:41:19.034361+0000","last_became_peered":"2024-09-15T03:41:19.034361+0000","last_unstale":"2024-09-15T03:42:07.075710+0000","last_undegraded":"2024-09-15T03:42:07.075710+0000","last_fullsized":"2024-09-15T03:42:07.075710+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:11.092582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010695812000000001,"stat_sum":{"num_bytes":9781248,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":10,"num_write":45,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,1,9,14,6,11],"acting":[8,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.b","version":"100'42","reported_seq":345,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.081270+0000","last_change":"2024-09-15T03:42:07.083548+0000","last_active":"2024-09-15T03:42:07.081270+0000","last_peered":"2024-09-15T03:42:07.081270+0000","last_clean":"2024-09-15T03:42:07.081270+0000","last_became_active":"2024-09-15T03:41:50.018104+0000","last_became_peered":"2024-09-15T03:41:50.018104+0000","last_unstale":"2024-09-15T03:42:07.081270+0000","last_undegraded":"2024-09-15T03:42:07.081270+0000","last_fullsized":"2024-09-15T03:42:07.081270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'40","last_scrub_stamp":"2024-09-15T03:42:06.615211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:42:06.615211+0000","objects_scrubbed":4,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.541627+0000","scrub_duration":0.0033695489999999999,"objects_trimmed":0,"snaptrim_duration":0.018796686,"stat_sum":{"num_bytes":4653056,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":12296,"num_write":40,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5,1,12,6,7,9],"acting":[14,5,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.c","version":"100'57","reported_seq":298,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.071107+0000","last_change":"2024-09-15T03:42:07.071701+0000","last_active":"2024-09-15T03:42:07.071107+0000","last_peered":"2024-09-15T03:42:07.071107+0000","last_clean":"2024-09-15T03:42:07.071107+0000","last_became_active":"2024-09-15T03:41:39.015749+0000","last_became_peered":"2024-09-15T03:41:39.015749+0000","last_unstale":"2024-09-15T03:42:07.071107+0000","last_undegraded":"2024-09-15T03:42:07.071107+0000","last_fullsized":"2024-09-15T03:42:07.071107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:11.128571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070989579999999998,"stat_sum":{"num_bytes":2244608,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":14508,"num_write":47,"num_write_kb":7008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,0,10,6,12,5],"acting":[14,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.d","version":"94'42","reported_seq":329,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.019201+0000","last_change":"2024-09-15T03:42:02.016295+0000","last_active":"2024-09-15T03:42:05.019201+0000","last_peered":"2024-09-15T03:42:05.019201+0000","last_clean":"2024-09-15T03:42:05.019201+0000","last_became_active":"2024-09-15T03:41:50.021249+0000","last_became_peered":"2024-09-15T03:41:50.021249+0000","last_unstale":"2024-09-15T03:42:05.019201+0000","last_undegraded":"2024-09-15T03:42:05.019201+0000","last_fullsized":"2024-09-15T03:42:05.019201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038017699999999999,"stat_sum":{"num_bytes":1982464,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10422,"num_write":24,"num_write_kb":7200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,14,10,9,8,2],"acting":[7,0,14,10,9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.e","version":"95'87","reported_seq":377,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.360189+0000","last_change":"2024-09-15T03:42:07.360129+0000","last_active":"2024-09-15T03:42:07.360189+0000","last_peered":"2024-09-15T03:42:07.360189+0000","last_clean":"2024-09-15T03:42:07.360189+0000","last_became_active":"2024-09-15T03:41:39.015840+0000","last_became_peered":"2024-09-15T03:41:39.015840+0000","last_unstale":"2024-09-15T03:42:07.360189+0000","last_undegraded":"2024-09-15T03:42:07.360189+0000","last_fullsized":"2024-09-15T03:42:07.360189+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'87","last_scrub_stamp":"2024-09-15T03:42:07.360044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.360044+0000","objects_scrubbed":3,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:11.109168+0000","scrub_duration":0.0027624279999999999,"objects_trimmed":0,"snaptrim_duration":0.00056164799999999999,"stat_sum":{"num_bytes":5685248,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":25407,"num_write":29,"num_write_kb":11328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,1,11,8,6,12],"acting":[2,10,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.f","version":"97'89","reported_seq":291,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027726+0000","last_change":"2024-09-15T03:42:02.027714+0000","last_active":"2024-09-15T03:42:05.027726+0000","last_peered":"2024-09-15T03:42:05.027726+0000","last_clean":"2024-09-15T03:42:05.027726+0000","last_became_active":"2024-09-15T03:40:59.584644+0000","last_became_peered":"2024-09-15T03:40:59.584644+0000","last_unstale":"2024-09-15T03:42:05.027726+0000","last_undegraded":"2024-09-15T03:42:05.027726+0000","last_fullsized":"2024-09-15T03:42:05.027726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:07.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001140117,"stat_sum":{"num_bytes":12206080,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":1978,"num_write":98,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.10","version":"95'69","reported_seq":290,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.070885+0000","last_change":"2024-09-15T03:42:07.074650+0000","last_active":"2024-09-15T03:42:07.070885+0000","last_peered":"2024-09-15T03:42:07.070885+0000","last_clean":"2024-09-15T03:42:07.070885+0000","last_became_active":"2024-09-15T03:41:50.019077+0000","last_became_peered":"2024-09-15T03:41:50.019077+0000","last_unstale":"2024-09-15T03:42:07.070885+0000","last_undegraded":"2024-09-15T03:42:07.070885+0000","last_fullsized":"2024-09-15T03:42:07.070885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'33","last_scrub_stamp":"2024-09-15T03:41:29.106893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:29.106893+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:49.054548+0000","scrub_duration":0.39459468800000003,"objects_trimmed":0,"snaptrim_duration":0.0048221890000000002,"stat_sum":{"num_bytes":4423680,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":579,"num_write":12,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,5,8,0,11],"acting":[13,6,7,5,8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.11","version":"71'31","reported_seq":275,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.069591+0000","last_change":"2024-09-15T03:42:07.070028+0000","last_active":"2024-09-15T03:42:07.069591+0000","last_peered":"2024-09-15T03:42:07.069591+0000","last_clean":"2024-09-15T03:42:07.069591+0000","last_became_active":"2024-09-15T03:41:17.030558+0000","last_became_peered":"2024-09-15T03:41:17.030558+0000","last_unstale":"2024-09-15T03:42:07.069591+0000","last_undegraded":"2024-09-15T03:42:07.069591+0000","last_fullsized":"2024-09-15T03:42:07.069591+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'17","last_scrub_stamp":"2024-09-15T03:41:17.535319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:17.535319+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:20.364025+0000","scrub_duration":0.003091599,"objects_trimmed":0,"snaptrim_duration":0.00056524600000000004,"stat_sum":{"num_bytes":3309568,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":5188,"num_write":7,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.12","version":"35'2","reported_seq":218,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-15T03:42:07.074859+0000","last_change":"2024-09-15T03:42:07.075474+0000","last_active":"2024-09-15T03:42:07.074859+0000","last_peered":"2024-09-15T03:42:07.074859+0000","last_clean":"2024-09-15T03:42:07.074859+0000","last_became_active":"2024-09-15T03:41:50.026297+0000","last_became_peered":"2024-09-15T03:41:50.026297+0000","last_unstale":"2024-09-15T03:42:07.074859+0000","last_undegraded":"2024-09-15T03:42:07.074859+0000","last_fullsized":"2024-09-15T03:42:07.074859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-15T03:41:29.466491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:29.466491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:36.023400+0000","scrub_duration":0.0021447580000000001,"objects_trimmed":0,"snaptrim_duration":0.000861165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,11,2,15],"acting":[9,7,8,6,11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"}]},{"pgid":"3.13","version":"94'64","reported_seq":385,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-15T03:42:05.027659+0000","last_change":"2024-09-15T03:42:02.027314+0000","last_active":"2024-09-15T03:42:05.027659+0000","last_peered":"2024-09-15T03:42:05.027659+0000","last_clean":"2024-09-15T03:42:05.027659+0000","last_became_active":"2024-09-15T03:41:50.024851+0000","last_became_peered":"2024-09-15T03:41:50.024851+0000","last_unstale":"2024-09-15T03:42:05.027659+0000","last_undegraded":"2024-09-15T03:42:05.027659+0000","last_fullsized":"2024-09-15T03:42:05.027659+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'21","last_scrub_stamp":"2024-09-15T03:41:19.809341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:19.809341+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:21.113176+0000","scrub_duration":0.0033873140000000002,"objects_trimmed":0,"snaptrim_duration":0.001305797,"stat_sum":{"num_bytes":3227648,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":18,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,5,2,0,15,10],"acting":[11,7,5,2,0,15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"25","length":"1"},{"start":"2e","length":"1"},{"start":"36","length":"1"},{"start":"38","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":88539136,"num_objects":54,"num_object_clones":13,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":3,"num_read":279,"num_read_kb":144702,"num_write":503,"num_write_kb":109408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":53411840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159145984,"data_stored":189804544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":0},"log_size":1056,"ondisk_log_size":1056,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280531,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313528,"kb_used_data":12008,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058312,"statfs":{"total":96636764160,"available":96315711488,"internally_reserved":0,"allocated":12296192,"data_stored":13461859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":170.74299999999999,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6849999999999996,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.501},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6980000000000004,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.85899999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.75900000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.77400000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":23.472999999999999,"5min":23.472999999999999,"15min":23.472999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":478.76900000000001,"5min":478.76900000000001,"15min":478.76900000000001},"last":0.996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.80000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":11,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.68100000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":1.012},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.878}]}]},{"osd":14,"up_from":22,"seq":94489280532,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315348,"kb_used_data":13828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056492,"statfs":{"total":96636764160,"available":96313847808,"internally_reserved":0,"allocated":14159872,"data_stored":15865667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.60199999999999998},{"interface":"front","average":{"1min":48.625999999999998,"5min":48.625999999999998,"15min":48.625999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":913.03200000000004,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.79100000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.629},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91600000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.754}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":30.189,"5min":30.189,"15min":30.189},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":560.85299999999995,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.85899999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":1.03},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":1.141}]}]},{"osd":13,"up_from":22,"seq":94489280531,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313776,"kb_used_data":12260,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058064,"statfs":{"total":96636764160,"available":96315457536,"internally_reserved":0,"allocated":12554240,"data_stored":14011280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.90400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":4,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.876}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.83099999999999996},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.81299999999999994}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.145}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":10,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":18.027999999999999,"5min":18.027999999999999,"15min":18.027999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":379.39100000000002,"5min":379.39100000000002,"15min":379.39100000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":41.061999999999998,"5min":41.061999999999998,"15min":41.061999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":886.51800000000003,"5min":886.51800000000003,"15min":886.51800000000003},"last":1.2090000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":1.1850000000000001},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":1.0309999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.288}]},{"osd":14,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":13.311,"5min":13.311,"15min":13.311},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":274.65100000000001,"5min":274.65100000000001,"15min":274.65100000000001},"last":1.2450000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":1.321}]}]},{"osd":0,"up_from":21,"seq":90194313237,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315124,"kb_used_data":13604,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056716,"statfs":{"total":96636764160,"available":96314077184,"internally_reserved":0,"allocated":13930496,"data_stored":18755720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":41.966000000000001,"5min":41.966000000000001,"15min":41.966000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":703.98400000000004,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.745}]},{"osd":7,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":25.172999999999998,"5min":25.172999999999998,"15min":25.172999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":419.01499999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.68200000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.39800000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70199999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":3.972,"5min":3.972,"15min":3.972},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":62.659999999999997,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.80100000000000005}]}]},{"osd":6,"up_from":21,"seq":90194313235,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315268,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056572,"statfs":{"total":96636764160,"available":96313929728,"internally_reserved":0,"allocated":14082048,"data_stored":15658520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":37.668999999999997,"5min":37.668999999999997,"15min":37.668999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":668.923,"5min":668.923,"15min":668.923},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51700000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":15.977,"5min":15.977,"15min":15.977},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":290.32400000000001,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.60099999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.58899999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70299999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.77200000000000002}]}]},{"osd":15,"up_from":21,"seq":90194313237,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306640,"kb_used_data":9276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065200,"statfs":{"total":96636764160,"available":96322764800,"internally_reserved":0,"allocated":9498624,"data_stored":11354867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":45.473999999999997,"5min":45.473999999999997,"15min":45.473999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":764.62199999999996,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.63100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68700000000000006}]},{"osd":5,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.442}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":18.594999999999999,"5min":18.594999999999999,"15min":18.594999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":305.18200000000002,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.54900000000000004}]},{"osd":7,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":58.883000000000003,"5min":58.883000000000003,"15min":58.883000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":782.05200000000002,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47699999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.67700000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.69599999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58099999999999996}]},{"osd":14,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65500000000000003}]}]},{"osd":3,"up_from":0,"seq":292057776137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":22,"seq":94489280531,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313432,"kb_used_data":11908,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058408,"statfs":{"total":96636764160,"available":96315809792,"internally_reserved":0,"allocated":12193792,"data_stored":12408117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.375},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.64200000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.58199999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":10,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.71299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63300000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":40.006999999999998,"5min":40.006999999999998,"15min":40.006999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":740.69799999999998,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.70299999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.66600000000000004}]}]},{"osd":11,"up_from":22,"seq":94489280531,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314072,"kb_used_data":12556,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94057768,"statfs":{"total":96636764160,"available":96315154432,"internally_reserved":0,"allocated":12857344,"data_stored":12394008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.58099999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":892.99000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.64000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.177,"5min":3.177,"15min":3.177},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.71599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":33.399999999999999,"5min":33.399999999999999,"15min":33.399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":557.40099999999995,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.77600000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.93100000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":41.654000000000003,"5min":41.654000000000003,"15min":41.654000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":694.31399999999996,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.91900000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":11.574,"5min":11.574,"15min":11.574},"last":0.81899999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.96099999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":12,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304840,"kb_used_data":7484,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94067000,"statfs":{"total":96636764160,"available":96324608000,"internally_reserved":0,"allocated":7663616,"data_stored":7601129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.318},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.43099999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":10.916,"5min":10.916,"15min":10.916},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":201.73599999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.54700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":14.640000000000001,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.39700000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":818.68700000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.56399999999999995}]},{"osd":15,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":34.192,"5min":34.192,"15min":34.192},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":703.52499999999998,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.55500000000000005}]}]},{"osd":1,"up_from":22,"seq":94489280531,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311848,"kb_used_data":14492,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94059992,"statfs":{"total":96636764160,"available":96317431808,"internally_reserved":0,"allocated":14839808,"data_stored":16931798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.44},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.34399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":10.869999999999999,"5min":10.869999999999999,"15min":10.869999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":214.91,"5min":214.91,"15min":214.91},"last":0.45600000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.624}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.46899999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.48899999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.61199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.010000000000002}]}]},{"osd":4,"up_from":0,"seq":94489280532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":22,"seq":94489280531,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313612,"kb_used_data":12092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058228,"statfs":{"total":96636764160,"available":96315625472,"internally_reserved":0,"allocated":12382208,"data_stored":15000852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.34899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.438},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.48999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.47499999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.53100000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.71099999999999997}]},{"osd":15,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.84299999999999997}]}]},{"osd":8,"up_from":22,"seq":94489280532,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":12912,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":13221888,"data_stored":13289283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.377}]},{"osd":1,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.503}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":903.59500000000003}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.57799999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":22.788,"5min":22.788,"15min":22.788},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":398.71699999999998,"5min":398.71699999999998,"15min":398.71699999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55600000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":51.093000000000004,"5min":51.093000000000004,"15min":51.093000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":908.44200000000001,"5min":908.44200000000001,"15min":908.44200000000001},"last":0.53200000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":8.952,"5min":8.952,"15min":8.952},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":149.09,"5min":149.09,"15min":149.09},"last":0.57099999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.62},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.629}]},{"osd":14,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.68999999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313300,"kb_used_data":11780,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058540,"statfs":{"total":96636764160,"available":96315944960,"internally_reserved":0,"allocated":12062720,"data_stored":13731651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":10.802,"5min":10.802,"15min":10.802},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":151.80600000000001,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.70599999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":7.327,"5min":7.327,"15min":7.327},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":98.858000000000004,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.88400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":37.173999999999999,"5min":37.173999999999999,"15min":37.173999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":14.319000000000001,"5min":14.319000000000001,"15min":14.319000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":203.91399999999999,"5min":203.91399999999999,"15min":203.91399999999999},"last":0.82099999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.94699999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.85399999999999998},{"interface":"front","average":{"1min":5.7759999999999998,"5min":5.7759999999999998,"15min":5.7759999999999998},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":73.846999999999994,"5min":73.846999999999994,"15min":73.846999999999994},"last":0.871}]},{"osd":15,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.80600000000000005}]}]},{"osd":10,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307872,"kb_used_data":10512,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94063968,"statfs":{"total":96636764160,"available":96321503232,"internally_reserved":0,"allocated":10764288,"data_stored":12314435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57299999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.65700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42399999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69899999999999995},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.70999999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":16.597000000000001,"5min":16.597000000000001,"15min":16.597000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":287.86900000000003,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12406784,"data_stored":17850368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13381632,"data_stored":16068608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":12058624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6684672,"data_stored":7659520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10903552,"data_stored":14127104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":14794752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11325440,"data_stored":12808192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11743232,"data_stored":12414976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10584064,"data_stored":12500992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9285632,"data_stored":11440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11395072,"data_stored":11530240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":6201344,"data_stored":6737920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":10567680,"data_stored":12619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12681216,"data_stored":14991360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8634368,"data_stored":11755520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:11.841 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-15T03:42:11.842 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-15T03:42:11.842 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-15T03:42:11.842 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-09-15T03:42:11.842 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=4, reqid=client.4562.0:46, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.828216052000002, 2024-09-15T03:42:11.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013955543000000001, 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031622+0000", 2024-09-15T03:42:11.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031623+0000", 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4900000000000005e-07 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031726+0000", 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103491 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031757+0000", 2024-09-15T03:42:11.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0908999999999998e-05 2024-09-15T03:42:11.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031770+0000", 2024-09-15T03:42:11.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2565000000000001e-05 2024-09-15T03:42:11.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031863+0000", 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2622999999999997e-05 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.031928+0000", 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5984999999999999e-05 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045547+0000", 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013618655 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045578+0000", 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0565999999999998e-05 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.as3 35/32 ECSubWrite(tid=5, reqid=client.4562.0:47, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:42:11.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.827252881000007, 2024-09-15T03:42:11.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013036834000000001, 2024-09-15T03:42:11.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.871 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:42:11.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032585+0000", 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032586+0000", 2024-09-15T03:42:11.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032605+0000", 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-05 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032648+0000", 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2763999999999999e-05 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032656+0000", 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0739999999999993e-06 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032693+0000", 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6701e-05 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.032699+0000", 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6849999999999998e-06 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045592+0000", 2024-09-15T03:42:11.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012893345 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.045622+0000", 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0182e-05 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=25, reqid=client.4562.0:101, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.775496532000005, 2024-09-15T03:42:11.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011334317, 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084342+0000", 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.084360+0000", 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7618999999999999e-05 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088093+0000", 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037333290000000001 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088098+0000", 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7010000000000004e-06 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088131+0000", 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2653000000000001e-05 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.088136+0000", 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4979999999999997e-06 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095632+0000", 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074962550000000003 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.095676+0000", 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3516999999999997e-05 2024-09-15T03:42:11.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.7s2 35/32 ECSubRead(tid=32, to_read={3:edf86ef6:::smithi10536496-37:head=0,4096,0}, subchunks={3:edf86ef6:::smithi10536496-37:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.754398166000001, 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039281828999999997, 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:42:11.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105440+0000", 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105441+0000", 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8500000000000001e-07 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.105447+0000", 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5010000000000001e-06 2024-09-15T03:42:11.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141752+0000", 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036305044000000002 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141762+0000", 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3940000000000002e-06 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.141935+0000", 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017359499999999999 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.144722+0000", 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00278671 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=34, reqid=client.4562.0:140, at_version=35'6, trim_to=0'0, roll_forward_to=35'3))", 2024-09-15T03:42:11.886 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.717325138999996, 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038895118999999999, 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142513+0000", 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142514+0000", 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3900000000000004e-07 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.142528+0000", 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.449e-05 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180934+0000", 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038406005999999999 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180939+0000", 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4270000000000001e-06 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180984+0000", 2024-09-15T03:42:11.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5751000000000003e-05 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.180995+0000", 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0332e-05 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181383+0000", 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038812500000000002 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.181408+0000", 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5349000000000001e-05 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=40, reqid=client.4562.0:176, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-15T03:42:11.892 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:42:11.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.665337847000004, 2024-09-15T03:42:11.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229890967, 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194500+0000", 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194501+0000", 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194514+0000", 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3422e-05 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423889+0000", 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.229374775 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423894+0000", 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0669999999999999e-06 2024-09-15T03:42:11.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423938+0000", 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3967000000000001e-05 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.423948+0000", 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2920000000000001e-06 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424370+0000", 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042234599999999998 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.424391+0000", 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1441999999999999e-05 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=75, reqid=client.4562.0:298, at_version=47'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:42:11.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.845576485999999, 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01362943, 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014262+0000", 2024-09-15T03:42:11.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014263+0000", 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0100000000000001e-06 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014284+0000", 2024-09-15T03:42:11.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1742e-05 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014322+0000", 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7746000000000001e-05 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014330+0000", 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2020000000000004e-06 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014378+0000", 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.799e-05 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.014389+0000", 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0265000000000001e-05 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027856+0000", 2024-09-15T03:42:11.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013467549000000001 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027891+0000", 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4925999999999997e-05 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/44 ECSubWrite(tid=76, reqid=client.4562.0:299, at_version=47'19, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.844501813000001, 2024-09-15T03:42:11.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01257076, 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015336+0000", 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015337+0000", 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.65e-07 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015351+0000", 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4504e-05 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015447+0000", 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5996999999999996e-05 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.015454+0000", 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4250000000000003e-06 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016973+0000", 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001519248 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.016980+0000", 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4449999999999999e-06 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027896+0000", 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010915784 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:14.027907+0000", 2024-09-15T03:42:11.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0992000000000001e-05 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:399 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.808260349999998, 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014049582999999999, 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 399 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051578+0000", 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051580+0000", 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.624e-06 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051584+0000", 2024-09-15T03:42:11.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3320000000000002e-06 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051594+0000", 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.027e-05 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051600+0000", 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3500000000000002e-06 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5226000000000003e-05 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051663+0000", 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.25e-06 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058108+0000", 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0064450269999999999 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.058282+0000", 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017381800000000001 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059091+0000", 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080964600000000002 2024-09-15T03:42:11.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062585+0000", 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003493875 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065538+0000", 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002952826 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.065627+0000", 2024-09-15T03:42:11.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339000000000006e-05 2024-09-15T03:42:11.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:400 3.15s0 3:ac8f3d24:::smithi10536496-17:head [rollback c] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.808223830000003, 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016394477000000001, 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 400 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051614+0000", 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051615+0000", 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051618+0000", 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.977e-06 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051623+0000", 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5010000000000004e-06 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051626+0000", 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8009999999999999e-06 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051729+0000", 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010270800000000001 2024-09-15T03:42:11.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051734+0000", 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9989999999999999e-06 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059522+0000", 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007788059 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.059592+0000", 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0457999999999995e-05 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060158+0000", 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056606199999999999 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062892+0000", 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002733769 2024-09-15T03:42:11.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.067964+0000", 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050720879999999998 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068009+0000", 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4685999999999999e-05 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.921 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:401 3.15s0 3:ac8f3d24:::smithi10536496-17:head [append 0~0] snapc 11=[10,c,b] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.808190136999997, 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016504504, 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 401 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051648+0000", 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0100000000000001e-07 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051651+0000", 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051656+0000", 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9250000000000003e-06 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051659+0000", 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1499999999999999e-06 2024-09-15T03:42:11.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051769+0000", 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011069 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.051774+0000", 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6430000000000003e-06 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060449+0000", 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.008674747 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060522+0000", 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3393999999999996e-05 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.060841+0000", 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031853799999999999 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.062921+0000", 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002080765 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068093+0000", 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051714500000000002 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:21.068153+0000", 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9673000000000003e-05 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs6 67/32 ECSubWrite(tid=116, reqid=client.4562.0:494, at_version=67'25, trim_to=0'0, roll_forward_to=65'24))", 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.807672576000002, 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37552997599999999, 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.052166+0000", 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-07 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.053528+0000", 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013622840000000001 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426011+0000", 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.37248246099999999 2024-09-15T03:42:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426019+0000", 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3089999999999996e-06 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426108+0000", 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9092999999999996e-05 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.426118+0000", 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0407e-05 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427667+0000", 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015489220000000001 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:34.427696+0000", 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8337999999999999e-05 2024-09-15T03:42:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:764 3.15s0 3:ac8f3d24:::smithi10536496-17:head [read 0~3489792,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:57.046754+0000", 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.813084036999999, 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011213247000000001, 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 764 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046754+0000", 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046754+0000", 2024-09-15T03:42:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046756+0000", 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1950000000000002e-06 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046767+0000", 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0954e-05 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046786+0000", 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8987999999999998e-05 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046793+0000", 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0940000000000002e-06 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046853+0000", 2024-09-15T03:42:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9726999999999999e-05 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046859+0000", 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8760000000000002e-06 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.048990+0000", 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002130921 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.049058+0000", 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8480999999999997e-05 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.057967+0000", 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089090109999999997 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:765 3.15s0 3:ac8f3d24:::smithi10536496-17:head [read 0~3489792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-15T03:42:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:57.046805+0000", 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.813032966, 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012962850999999999, 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 765 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046805+0000", 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046805+0000", 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046806+0000", 2024-09-15T03:42:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046810+0000", 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.206e-06 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046820+0000", 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0373999999999999e-05 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046824+0000", 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4300000000000002e-06 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046939+0000", 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114819 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046946+0000", 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7009999999999996e-06 2024-09-15T03:42:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.049433+0000", 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024869610000000002 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.049520+0000", 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6417999999999998e-05 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.059768+0000", 2024-09-15T03:42:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010248345000000001 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:766 3.15s0 3:ac8f3d24:::smithi10536496-17:head [read 0~3489792,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:57.046843+0000", 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.812995151999999, 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014797642999999999, 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 766 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046843+0000", 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046843+0000", 2024-09-15T03:42:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046844+0000", 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0299999999999999e-07 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046847+0000", 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5370000000000002e-06 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046856+0000", 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5639999999999997e-06 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046861+0000", 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1590000000000001e-06 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046975+0000", 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011406899999999999 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.046996+0000", 2024-09-15T03:42:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1467e-05 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.049939+0000", 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002942624 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.050043+0000", 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103803 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:57.061641+0000", 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011597916999999999 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:786 3.15s0 3:ac8f3d24:::smithi10536496-17:head [delete] snapc 38=[36] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94)", 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:01.018028+0000", 2024-09-15T03:42:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.841810233, 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015959363000000001, 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 786 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.018028+0000", 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.018028+0000", 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.018030+0000", 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9929999999999998e-06 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.018035+0000", 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.242e-06 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.018053+0000", 2024-09-15T03:42:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8003999999999999e-05 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.018061+0000", 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.418e-06 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.019234+0000", 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011738180000000001 2024-09-15T03:42:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.019362+0000", 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012800499999999999 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.020102+0000", 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073950600000000002 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.023812+0000", 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037101030000000002 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.033917+0000", 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010105385999999999 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:01.033987+0000", 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9888000000000006e-05 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:788 3.15s0 3:ac8f3d24:::smithi10536496-17:head [copy-from ver 32] snapc 38=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:02.017414+0000", 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8424247200000003, 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022659756, 2024-09-15T03:42:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 788 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017414+0000", 2024-09-15T03:42:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017414+0000", 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017416+0000", 2024-09-15T03:42:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.932e-06 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017425+0000", 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6640000000000004e-06 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017445+0000", 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9854999999999999e-05 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017452+0000", 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0899999999999999e-06 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017521+0000", 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9367000000000002e-05 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017531+0000", 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3919999999999992e-06 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.020032+0000", 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002501053 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.020155+0000", 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012303300000000001 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.034822+0000", 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014667065999999999 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.038397+0000", 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035749750000000002 2024-09-15T03:42:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.039995+0000", 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015982310000000001 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.040073+0000", 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8097999999999997e-05 2024-09-15T03:42:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:789 3.15s0 3:ac8f3d24:::smithi10536496-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:02.017465+0000", 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8423736789999996, 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020227028000000001, 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 789 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017465+0000", 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017465+0000", 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017465+0000", 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017470+0000", 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0760000000000002e-06 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017476+0000", 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4419999999999997e-06 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017478+0000", 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.756e-06 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017592+0000", 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113294 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.017602+0000", 2024-09-15T03:42:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9669999999999996e-06 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.020340+0000", 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027381530000000001 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for blocked object", 2024-09-15T03:42:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.020377+0000", 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7188999999999999e-05 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.037472+0000", 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017094917000000001 2024-09-15T03:42:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.037566+0000", 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3708000000000005e-05 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:02.037692+0000", 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012599700000000001 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:920 3.15s0 3:ac8f3d24:::smithi10536496-17:head [copy-from ver 10] snapc 44=[44,43,42,41,3f,3e,3c] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:11.018697+0000", 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.84114151100000001, 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025154853000000001, 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 920 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018697+0000", 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018697+0000", 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018698+0000", 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018707+0000", 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9250000000000001e-06 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018726+0000", 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8553e-05 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018734+0000", 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9740000000000002e-06 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018795+0000", 2024-09-15T03:42:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1161000000000006e-05 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018803+0000", 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9000000000000006e-06 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.020417+0000", 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001614489 2024-09-15T03:42:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.020530+0000", 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113052 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.032630+0000", 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012099732 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.035364+0000", 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027343189999999998 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.043807+0000", 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084427419999999996 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.043852+0000", 2024-09-15T03:42:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4533000000000002e-05 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:921 3.15s0 3:ac8f3d24:::smithi10536496-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:11.018749+0000", 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.841089219, 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015997526000000001, 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 921 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018749+0000", 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018749+0000", 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018750+0000", 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018753+0000", 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9730000000000002e-06 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018761+0000", 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4829999999999999e-06 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018766+0000", 2024-09-15T03:42:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8439999999999997e-06 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018845+0000", 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9097999999999994e-05 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.018849+0000", 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5559999999999999e-06 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.020686+0000", 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018376009999999999 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for blocked object", 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.020760+0000", 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3887000000000006e-05 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.034526+0000", 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013766364 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.034636+0000", 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010988399999999999 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:11.034747+0000", 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110299 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:11.989 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:11.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-15T03:42:11.992 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:11.989+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-15T03:42:12.027 INFO:tasks.rados.rados.0.smithi105.stdout:378: done (3 left) 2024-09-15T03:42:12.027 INFO:tasks.rados.rados.0.smithi105.stdout:379: done (2 left) 2024-09-15T03:42:12.027 INFO:tasks.rados.rados.0.smithi105.stdout:380: done (1 left) 2024-09-15T03:42:12.027 INFO:tasks.rados.rados.0.smithi105.stdout:381: done (0 left) 2024-09-15T03:42:12.028 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 17 current snap is 38 2024-09-15T03:42:12.028 INFO:tasks.rados.rados.0.smithi105.stdout:382: seq_num 148 ranges {1818624=81920} 2024-09-15T03:42:12.037 INFO:tasks.rados.rados.0.smithi105.stdout:382: writing smithi10536496-17 from 1818624 to 1900544 tid 1 2024-09-15T03:42:12.037 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 39 current snap is 38 2024-09-15T03:42:12.038 INFO:tasks.rados.rados.0.smithi105.stdout:383: seq_num 149 ranges {2424832=98304} 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:383: writing smithi10536496-39 from 2424832 to 2523136 tid 1 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:384: copy_from oid 46 from oid 10 current snap is 38 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:382: finishing write tid 1 to smithi10536496-17 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:382: finishing write tid 2 to smithi10536496-17 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:382: finishing write tid 3 to smithi10536496-17 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 54 (ObjNum 148 snap 38 seq_num 148) dirty exists 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:382: left oid 17 (ObjNum 148 snap 38 seq_num 148) 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:382: done (2 left) 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:385: read oid 9 snap 37 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:385: expect deleted 2024-09-15T03:42:12.052 INFO:tasks.rados.rados.0.smithi105.stdout:386: delete oid 14 current snap is 38 2024-09-15T03:42:12.058 INFO:tasks.rados.rados.0.smithi105.stdout:383: finishing write tid 1 to smithi10536496-39 2024-09-15T03:42:12.059 INFO:tasks.rados.rados.0.smithi105.stdout:383: finishing write tid 2 to smithi10536496-39 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:385: done (3 left) 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:386: done (2 left) 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:387: read oid 43 snap -1 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:387: expect deleted 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:388: read oid 14 snap -1 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:388: expect deleted 2024-09-15T03:42:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:389: read oid 45 snap -1 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:389: expect deleted 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:390: read oid 48 snap 34 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:390: expect (ObjNum 114 snap 22 seq_num 114) 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:391: delete oid 20 current snap is 38 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:383: finishing write tid 3 to smithi10536496-39 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 64 (ObjNum 149 snap 38 seq_num 149) dirty exists 2024-09-15T03:42:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:383: left oid 39 (ObjNum 149 snap 38 seq_num 149) 2024-09-15T03:42:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:390: expect (ObjNum 114 snap 22 seq_num 114) 2024-09-15T03:42:12.091 INFO:tasks.ceph.osd.15.smithi105.stderr:2024-09-15T03:42:12.090+0000 7fc28f7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33111) UID: 0 2024-09-15T03:42:12.122 INFO:tasks.rados.rados.0.smithi105.stdout:384: finishing copy_from racing read to smithi10536496-46 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:932 3.18s0 3:1c642a04:::smithi10536496-46:head [copy-from ver 73 in=55b] snapc 45=[45,44,43,42,41,3f,3e,3c] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:12.051586+0000", 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.094104455000000004, 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094128930999999999, 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 932 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051586+0000", 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051586+0000", 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051587+0000", 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3510000000000001e-06 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051611+0000", 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3555000000000001e-05 2024-09-15T03:42:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051626+0000", 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5193e-05 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051630+0000", 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.476e-06 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051686+0000", 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5758999999999999e-05 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051859+0000", 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017321700000000001 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.080970+0000", 2024-09-15T03:42:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029110932999999999 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.086411+0000", 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054411440000000002 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-15T03:42:12.151 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:12.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-15T03:42:12.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-15T03:42:12.198 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:42:12.197+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:42:12.255 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-15T03:42:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-15T03:42:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-15T03:42:12.255 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:12.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-15T03:42:12.305 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:12.304+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:12.394 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:12.394+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.556750726000004, 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013303176, 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 51 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849454+0000", 2024-09-15T03:42:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9800000000000003e-07 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849459+0000", 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7999999999999998e-06 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849468+0000", 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4989999999999994e-06 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849473+0000", 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9869999999999999e-06 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855455+0000", 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059819499999999998 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855531+0000", 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6828000000000004e-05 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.855671+0000", 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013914 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860865+0000", 2024-09-15T03:42:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051942840000000004 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862666+0000", 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001801588 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862719+0000", 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3001999999999998e-05 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862757+0000", 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7299999999999999e-05 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.556722621000006, 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013362720999999999, 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:42:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 52 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849482+0000", 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4300000000000003e-07 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849488+0000", 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7180000000000001e-06 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849496+0000", 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7179999999999996e-06 2024-09-15T03:42:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.849503+0000", 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.81e-06 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856022+0000", 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065196560000000004 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856088+0000", 2024-09-15T03:42:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5982000000000006e-05 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.856175+0000", 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6425e-05 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.860870+0000", 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046953439999999997 2024-09-15T03:42:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862797+0000", 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019265949999999999 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862824+0000", 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7715000000000001e-05 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:43.862844+0000", 2024-09-15T03:42:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0214999999999999e-05 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4185.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.261881514999999, 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017810172999999999, 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4185", 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/3036622776", 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 100 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144323+0000", 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144326+0000", 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5320000000000001e-06 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144330+0000", 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5260000000000004e-06 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.144332+0000", 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0209999999999998e-06 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154403+0000", 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010071256000000001 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154449+0000", 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5555e-05 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.154520+0000", 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1303999999999994e-05 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.158204+0000", 2024-09-15T03:42:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003683258 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162087+0000", 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038832440000000001 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162113+0000", 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6605e-05 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:40:44.162133+0000", 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9562999999999999e-05 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=8, to_read={3:fff44eab:::smithi10536496-10:head=0,4096,0}, subchunks={3:fff44eab:::smithi10536496-10:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.372305170999994, 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046423195, 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033899+0000", 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033900+0000", 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000004e-07 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.033906+0000", 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080075+0000", 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046169038000000003 2024-09-15T03:42:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080080+0000", 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080125+0000", 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5151000000000002e-05 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.080322+0000", 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019686500000000001 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 35/32 ECSubWriteReply(tid=118, last_complete=35'2, committed=1, applied=1))", 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.319372888999993, 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021508349, 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086831+0000", 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086832+0000", 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2900000000000003e-07 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.086835+0000", 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0479999999999999e-06 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108275+0000", 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021439848000000001 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108283+0000", 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8609999999999998e-06 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108320+0000", 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7330999999999999e-05 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108340+0000", 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9531999999999999e-05 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 35/32 ECSubWrite(tid=17, reqid=client.4562.0:119, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.297799522999995, 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041863826999999999, 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108405+0000", 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.108419+0000", 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149106+0000", 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040686870999999999 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149110+0000", 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.335e-06 2024-09-15T03:42:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149157+0000", 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7131e-05 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.149164+0000", 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9630000000000001e-06 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150245+0000", 2024-09-15T03:42:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010804580000000001 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.150269+0000", 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4128e-05 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:132 3.3s0 3:cf5921a8:::smithi10536496-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.280990145999994, 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015465478, 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 132 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125214+0000", 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125215+0000", 2024-09-15T03:42:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125216+0000", 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7349999999999999e-06 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125223+0000", 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.125227+0000", 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8840000000000004e-06 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129903+0000", 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046767349999999996 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.129995+0000", 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1706000000000006e-05 2024-09-15T03:42:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133187+0000", 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031916510000000002 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.133234+0000", 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7862999999999999e-05 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.140680+0000", 2024-09-15T03:42:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074451860000000003 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=25, to_read={3:fc38c57e:::smithi10536496-49:head=0,4096,0}, subchunks={3:fc38c57e:::smithi10536496-49:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.249749383999998, 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038157982, 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156455+0000", 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-09-15T03:42:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.156460+0000", 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5569999999999997e-06 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194316+0000", 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037855609999999998 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194320+0000", 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999998e-06 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194361+0000", 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0708999999999999e-05 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.194613+0000", 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025227700000000002 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.fs3 35/32 ECSubRead(tid=31, to_read={3:f17f8228:::smithi10536496-40:head=0,167936,0}, subchunks={3:f17f8228:::smithi10536496-40:head=[0,1]}, attrs_to_read=))", 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.210301193999996, 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066290004, 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195903+0000", 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195904+0000", 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4899999999999995e-07 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.195911+0000", 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3030000000000002e-06 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262021+0000", 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066109629000000003 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262025+0000", 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5909999999999998e-06 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262085+0000", 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0183999999999998e-05 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:02.262193+0000", 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107648 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:277 3.13s0 3:ca2bd2fa:::smithi10536496-45:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.376650155999997, 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0157117019999999, 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 277 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029554+0000", 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029556+0000", 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.542e-06 2024-09-15T03:42:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029563+0000", 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2989999999999999e-06 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029577+0000", 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029601+0000", 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3200000000000001e-05 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029661+0000", 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0633e-05 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:11.029665+0000", 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5990000000000002e-06 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037509+0000", 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0078444609999999 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.037649+0000", 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000139476 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:12.045266+0000", 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076171729999999997 2024-09-15T03:42:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:305 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.401712281000002, 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0410521429999999, 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 305 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004492+0000", 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004494+0000", 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1660000000000001e-06 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004502+0000", 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1540000000000002e-06 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004520+0000", 2024-09-15T03:42:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7742000000000001e-05 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004530+0000", 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004681+0000", 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015092999999999999 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004692+0000", 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1232e-05 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009082+0000", 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043904649999999996 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009087+0000", 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7339999999999999e-06 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.033828+0000", 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0247413160000001 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.034013+0000", 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018443999999999999 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045544+0000", 2024-09-15T03:42:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011531351 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:306 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.401616101999998, 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0413215819999999, 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 306 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004588+0000", 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004590+0000", 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3260000000000001e-06 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004596+0000", 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010000000000002e-06 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004613+0000", 2024-09-15T03:42:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6892999999999999e-05 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004623+0000", 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268000000000001e-05 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009818+0000", 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051952880000000002 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009821+0000", 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0189999999999998e-06 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035613+0000", 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.025791608 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035700+0000", 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7298000000000003e-05 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.045910+0000", 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010209480999999999 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:307 3.10s0 3:0e027997:::smithi10536496-12:head [read 0~196608,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.401572188000003, 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.041622225, 2024-09-15T03:42:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 307 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004632+0000", 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004633+0000", 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0200000000000001e-07 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004639+0000", 2024-09-15T03:42:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5599999999999999e-06 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004649+0000", 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3319999999999998e-06 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.004656+0000", 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8870000000000003e-06 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051891749999999999 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:16.009845+0000", 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7899999999999998e-07 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.035995+0000", 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026149322 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.036088+0000", 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3162e-05 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.046254+0000", 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010166405999999999 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:335 3.10s0 3:092ceaee:::smithi10536496-5:head [copy-from ver 4] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.285404149999998, 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077783689000000003, 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 335 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120800+0000", 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120802+0000", 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4699999999999999e-06 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120808+0000", 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2999999999999998e-06 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120821+0000", 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-15T03:42:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120826+0000", 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7879999999999997e-06 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.120884+0000", 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8034e-05 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.121005+0000", 2024-09-15T03:42:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120637 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.130702+0000", 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096976079999999999 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.132697+0000", 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019945800000000001 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198536+0000", 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065839133999999994 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:17.198584+0000", 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7796000000000001e-05 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:369 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~589824] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.380925771999998, 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022704654000000001, 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 369 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025279+0000", 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.025281+0000", 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4159999999999998e-06 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028105+0000", 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028242380000000002 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028126+0000", 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0939999999999999e-05 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.028134+0000", 2024-09-15T03:42:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7649999999999993e-06 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032973+0000", 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048395670000000003 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.033153+0000", 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017928700000000001 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034716+0000", 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001562873 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047854+0000", 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013138212999999999 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047923+0000", 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9306000000000004e-05 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047983+0000", 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0049000000000001e-05 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:370 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~573440] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.373240473999999, 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018633602999999999, 2024-09-15T03:42:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 370 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032964+0000", 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.032966+0000", 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034471+0000", 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015050790000000001 2024-09-15T03:42:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034522+0000", 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0529999999999999e-05 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034528+0000", 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1750000000000002e-06 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041526+0000", 2024-09-15T03:42:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006997886 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.041652+0000", 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126415 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.042320+0000", 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066831300000000002 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051454+0000", 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091331659999999999 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051523+0000", 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9029999999999995e-05 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.051597+0000", 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4819000000000003e-05 2024-09-15T03:42:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:371 3.18s0 3:1c642a04:::smithi10536496-46:head [append 0~196608] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.371666331999997, 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827797000000001, 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 371 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034538+0000", 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034539+0000", 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.116e-06 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034715+0000", 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017624599999999999 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034727+0000", 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1912999999999999e-05 2024-09-15T03:42:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034736+0000", 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6270000000000006e-06 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044278+0000", 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095422289999999993 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.044402+0000", 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124227 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.045119+0000", 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071647099999999999 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054221+0000", 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091019350000000002 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054291+0000", 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0148999999999995e-05 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054366+0000", 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4883999999999997e-05 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:372 3.18s0 3:1c642a04:::smithi10536496-46:head [setxattr _header (56) in=7b] snapc e=[c,b] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.371459817999998, 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020478778, 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 372 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034745+0000", 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4199999999999995e-07 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034749+0000", 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1690000000000002e-06 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034759+0000", 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4409999999999999e-06 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034769+0000", 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0064999999999999e-05 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047175+0000", 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012405657 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047305+0000", 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013068800000000001 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.047652+0000", 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034670499999999999 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.054413+0000", 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067614540000000001 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055174+0000", 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076077999999999996 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055223+0000", 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9076999999999998e-05 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:373 3.18s0 3:1c642a04:::smithi10536496-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.371429890000002, 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021752218, 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 373 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034774+0000", 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034775+0000", 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1900000000000003e-07 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034778+0000", 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9249999999999999e-06 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034785+0000", 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5669999999999997e-06 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.034790+0000", 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.51e-06 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048073+0000", 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013283497 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.048128+0000", 2024-09-15T03:42:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4455e-05 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055532+0000", 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074043889999999999 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.055612+0000", 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9971999999999999e-05 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:41:18.056527+0000", 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091438399999999997 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4562.0:933 3.18s0 3:1c642a04:::smithi10536496-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-15T03:42:12.051640+0000", 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.35456414000000003, 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033672299000000003, 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-15T03:42:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4562", 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.105:0/838367110", 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 933 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051640+0000", 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-15T03:42:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051640+0000", 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051641+0000", 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5900000000000003e-07 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-15T03:42:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051642+0000", 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6759999999999999e-06 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051646+0000", 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2359999999999998e-06 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.051649+0000", 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3690000000000001e-06 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.052006+0000", 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035764100000000001 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for blocked object", 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.052062+0000", 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5106999999999999e-05 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.085100+0000", 2024-09-15T03:42:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033037994000000001 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.085208+0000", 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108045 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-15T03:42:12.085312+0000", 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010487199999999999 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-15T03:42:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-15T03:42:12.516 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-15T03:42:12.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:12.462+0000 7ff2fe59c640 0 --2- 172.21.15.5:0/3986888459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3000c36f0 0x7ff300168a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:42:12.517 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:42:12.518 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:42:12.518 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:42:12.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-15T03:42:12.519 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:12.516+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:12.519 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:42:12.529 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:42:12.529 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:42:12.550 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:42:12.550 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:42:12.569 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:42:12.569 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:42:12.589 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:42:12.589 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:42:12.609 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:42:12.610 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:42:12.618 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:42:12.618+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:42:12.630 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:42:12.630 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:42:12.635 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:12.636 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:12.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-15T03:42:12.648 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:42:12.649 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:42:12.669 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:42:12.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:42:12.689 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:42:12.689 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:42:12.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:42:12.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:42:12.719 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:12.718+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:12.729 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:12.734+0000 7ff2fe59c640 0 -- 172.21.15.5:0/3986888459 >> [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] conn(0x7ff30018aee0 msgr2=0x7ff30018d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:12.747 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:12.747 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:12.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-15T03:42:12.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-15T03:42:12.774 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:12.774 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:42:12.795 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:12.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:12.819 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-15T03:42:12.819+0000 7fab1b9f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33249) UID: 0 2024-09-15T03:42:12.854 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:12.854 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:12.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-15T03:42:12.921 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:12.921+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:12.959 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:12.960 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:12.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-15T03:42:12.981 INFO:tasks.rados.rados.0.smithi105.stdout:384: finishing copy_from to smithi10536496-46 2024-09-15T03:42:12.981 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 73 (ObjNum 125 snap 27 seq_num 125) dirty exists 2024-09-15T03:42:13.023 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:13.021+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:13.065 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.065 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-15T03:42:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.5 primary-affinity to 0 (802) 2024-09-15T03:42:13.123 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:13.122+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:13.172 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.172 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.181 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-15T03:42:13.224 INFO:tasks.ceph.osd.9.smithi064.stderr:2024-09-15T03:42:13.224+0000 7fcb44bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33254) UID: 0 2024-09-15T03:42:13.278 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.278 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-15T03:42:13.326 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:13.325+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:13.385 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.385 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.393 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-15T03:42:13.438 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:13.437+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:13.490 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.490 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-15T03:42:13.510 INFO:tasks.rados.rados.0.smithi105.stdout:383: done (6 left) 2024-09-15T03:42:13.510 INFO:tasks.rados.rados.0.smithi105.stdout:384: done (5 left) 2024-09-15T03:42:13.511 INFO:tasks.rados.rados.0.smithi105.stdout:387: done (4 left) 2024-09-15T03:42:13.511 INFO:tasks.rados.rados.0.smithi105.stdout:388: done (3 left) 2024-09-15T03:42:13.511 INFO:tasks.rados.rados.0.smithi105.stdout:389: done (2 left) 2024-09-15T03:42:13.511 INFO:tasks.rados.rados.0.smithi105.stdout:390: done (1 left) 2024-09-15T03:42:13.511 INFO:tasks.rados.rados.0.smithi105.stdout:391: done (0 left) 2024-09-15T03:42:13.511 INFO:tasks.rados.rados.0.smithi105.stdout:392: setattr oid 6 current snap is 38 2024-09-15T03:42:13.512 INFO:tasks.rados.rados.0.smithi105.stdout:393: rollback oid 23 current snap is 38 2024-09-15T03:42:13.512 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 23 to 36 2024-09-15T03:42:13.512 INFO:tasks.rados.rados.0.smithi105.stdout:394: read oid 44 snap -1 2024-09-15T03:42:13.513 INFO:tasks.rados.rados.0.smithi105.stdout:394: expect (ObjNum 89 snap 14 seq_num 89) 2024-09-15T03:42:13.513 INFO:tasks.rados.rados.0.smithi105.stdout:395: delete oid 48 current snap is 38 2024-09-15T03:42:13.518 INFO:tasks.rados.rados.0.smithi105.stdout:393: finishing rollback tid 0 to smithi10536496-23 2024-09-15T03:42:13.519 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 62 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-15T03:42:13.521 INFO:tasks.rados.rados.0.smithi105.stdout:393: finishing rollback tid 1 to smithi10536496-23 2024-09-15T03:42:13.523 INFO:tasks.rados.rados.0.smithi105.stdout:392: done (3 left) 2024-09-15T03:42:13.523 INFO:tasks.rados.rados.0.smithi105.stdout:395: done (2 left) 2024-09-15T03:42:13.524 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 13 current snap is 38 2024-09-15T03:42:13.524 INFO:tasks.rados.rados.0.smithi105.stdout:396: seq_num 153 ranges {2162688=163840} 2024-09-15T03:42:13.527 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:13.526+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:13.536 INFO:tasks.rados.rados.0.smithi105.stdout:396: writing smithi10536496-13 from 2162688 to 2326528 tid 1 2024-09-15T03:42:13.536 INFO:tasks.rados.rados.0.smithi105.stdout:397: copy_from oid 10 from oid 9 current snap is 38 2024-09-15T03:42:13.536 INFO:tasks.rados.rados.0.smithi105.stdout:394: expect (ObjNum 89 snap 14 seq_num 89) 2024-09-15T03:42:13.628 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:13.628+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:13.637 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.637 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-15T03:42:13.656 INFO:tasks.rados.rados.0.smithi105.stdout:397: finishing copy_from to smithi10536496-10 2024-09-15T03:42:13.656 INFO:tasks.rados.rados.0.smithi105.stdout:397: got expected ENOENT (src dne) 2024-09-15T03:42:13.657 INFO:tasks.rados.rados.0.smithi105.stdout:397: finishing copy_from racing read to smithi10536496-10 2024-09-15T03:42:13.657 INFO:tasks.rados.rados.0.smithi105.stdout:396: finishing write tid 1 to smithi10536496-13 2024-09-15T03:42:13.657 INFO:tasks.rados.rados.0.smithi105.stdout:396: finishing write tid 2 to smithi10536496-13 2024-09-15T03:42:13.657 INFO:tasks.rados.rados.0.smithi105.stdout:396: finishing write tid 3 to smithi10536496-13 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 65 (ObjNum 153 snap 38 seq_num 153) dirty exists 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:396: left oid 13 (ObjNum 153 snap 38 seq_num 153) 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:394: done (3 left) 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:396: done (2 left) 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:397: done (1 left) 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:append oid 17 current snap is 38 2024-09-15T03:42:13.658 INFO:tasks.rados.rados.0.smithi105.stdout:398: seq_num 154 ranges {1900544=180224} 2024-09-15T03:42:13.667 INFO:tasks.rados.rados.0.smithi105.stdout:398: writing smithi10536496-17 from 1900544 to 2080768 tid 1 2024-09-15T03:42:13.667 INFO:tasks.rados.rados.0.smithi105.stdout:399: read oid 30 snap 33 2024-09-15T03:42:13.667 INFO:tasks.rados.rados.0.smithi105.stdout:399: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:42:13.667 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-09-15T03:42:13.670 INFO:tasks.rados.rados.0.smithi105.stdout:399: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-15T03:42:13.678 INFO:tasks.rados.rados.0.smithi105.stdout:393: finishing rollback tid 2 to smithi10536496-23 2024-09-15T03:42:13.678 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 58 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-15T03:42:13.678 INFO:tasks.rados.rados.0.smithi105.stdout:393: done (2 left) 2024-09-15T03:42:13.678 INFO:tasks.rados.rados.0.smithi105.stdout:399: done (1 left) 2024-09-15T03:42:13.678 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-09-15T03:42:13.729 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-15T03:42:13.728+0000 7fbcc5bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33080) UID: 0 2024-09-15T03:42:13.742 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.742 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-15T03:42:13.830 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:13.829+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:13.847 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.847 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-15T03:42:13.873 INFO:tasks.rados.rados.0.smithi105.stdout:398: finishing write tid 1 to smithi10536496-17 2024-09-15T03:42:13.874 INFO:tasks.rados.rados.0.smithi105.stdout:398: finishing write tid 2 to smithi10536496-17 2024-09-15T03:42:13.931 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:13.931+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:13.954 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:13.955 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:13.963 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-15T03:42:14.023 INFO:tasks.rados.rados.0.smithi105.stdout:398: finishing write tid 3 to smithi10536496-17 2024-09-15T03:42:14.023 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 60 (ObjNum 154 snap 38 seq_num 154) dirty exists 2024-09-15T03:42:14.023 INFO:tasks.rados.rados.0.smithi105.stdout:398: left oid 17 (ObjNum 154 snap 38 seq_num 154) 2024-09-15T03:42:14.023 INFO:tasks.rados.rados.0.smithi105.stdout:398: done (0 left) 2024-09-15T03:42:14.026 INFO:tasks.rados.rados.0.smithi105.stderr:0 errors. 2024-09-15T03:42:14.027 INFO:tasks.rados.rados.0.smithi105.stderr: 2024-09-15T03:42:14.029 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:42:14.035 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-15T03:42:14.032+0000 7fb447df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33087) UID: 0 2024-09-15T03:42:14.040 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:42:14.105 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.105 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-15T03:42:14.133 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:14.132+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:14.233 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-15T03:42:14.233+0000 7f34331f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33085) UID: 0 2024-09-15T03:42:14.250 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.251 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.259 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-15T03:42:14.335 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:14.334+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:14.338 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:14.338 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":107,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:42:14.008792+0000","last_up_change":"2024-09-15T03:41:35.013591+0000","last_in_change":"2024-09-15T03:41:46.112015+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:40:57.987730+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"44","auid":0,"snap_mode":"selfmanaged","snap_seq":69,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":46,"up_from":68,"up_thru":46,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43941},{"type":"v1","addr":"172.21.15.105:6803","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43941},{"type":"v1","addr":"172.21.15.105:6805","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43941},{"type":"v1","addr":"172.21.15.105:6809","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43941},{"type":"v1","addr":"172.21.15.105:6807","nonce":43941}]},"public_addr":"172.21.15.105:6803/43941","cluster_addr":"172.21.15.105:6805/43941","heartbeat_back_addr":"172.21.15.105:6809/43941","heartbeat_front_addr":"172.21.15.105:6807/43941","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":0.076263427734375,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:49.977726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.988005+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:46.256258+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-15T03:42:08.516008+0000","dead_epoch":47},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-15T03:42:09.575529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:43.785792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:04.563391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.889587+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:52.239895+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:02.352976+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:46.547814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:41.474463+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:55.629737+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:40.572229+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.006408+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:51.897727+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":8,"to":14},{"from":15,"to":12}]},{"pgid":"3.b","mappings":[{"from":0,"to":5}]},{"pgid":"3.c","mappings":[{"from":11,"to":14}]},{"pgid":"3.e","mappings":[{"from":9,"to":10}]},{"pgid":"3.13","mappings":[{"from":9,"to":5},{"from":13,"to":2}]},{"pgid":"3.18","mappings":[{"from":6,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":61,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:42:14.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:14.357 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.357 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.366 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-15T03:42:14.435 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-15T03:42:14.435+0000 7febe1ff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33079) UID: 0 2024-09-15T03:42:14.463 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.463 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-15T03:42:14.536 INFO:tasks.ceph.osd.13.smithi064.stderr:2024-09-15T03:42:14.536+0000 7fa7d0df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33256) UID: 0 2024-09-15T03:42:14.568 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.569 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.576 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-15T03:42:14.637 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T03:42:14.636+0000 7f7a7b7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32799) UID: 0 2024-09-15T03:42:14.674 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:14.675 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:14.677 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.677 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.683 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-15T03:42:14.687 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-15T03:42:14.031512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105136691,"num_objects":75,"num_object_clones":28,"num_object_copies":505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":5,"num_read":397,"num_read_kb":172999,"num_write":726,"num_write_kb":127714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":53411859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1223,"ondisk_log_size":1223,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":4406072,"kb_used_data":193112,"kb_used_omap":23,"kb_used_meta":4212712,"kb_avail":1316799688,"statfs":{"total":1352914698240,"available":1348402880512,"internally_reserved":0,"allocated":197746688,"data_stored":260828629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23856,"internal_metadata":4313817808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18366464,"num_objects":18,"num_object_clones":16,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":33,"num_read_kb":28225,"num_write":81,"num_write_kb":17936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000692"},"pg_stats":[{"pgid":"3.19","version":"101'29","reported_seq":304,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.024131+0000","last_change":"2024-09-15T03:42:07.066744+0000","last_active":"2024-09-15T03:42:11.024131+0000","last_peered":"2024-09-15T03:42:11.024131+0000","last_clean":"2024-09-15T03:42:11.024131+0000","last_became_active":"2024-09-15T03:41:50.026750+0000","last_became_peered":"2024-09-15T03:41:50.026750+0000","last_unstale":"2024-09-15T03:42:11.024131+0000","last_undegraded":"2024-09-15T03:42:11.024131+0000","last_fullsized":"2024-09-15T03:42:11.024131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'21","last_scrub_stamp":"2024-09-15T03:41:30.737422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.737422+0000","objects_scrubbed":4,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:59.283558+0000","scrub_duration":0.0027345220000000001,"objects_trimmed":0,"snaptrim_duration":0.001624871,"stat_sum":{"num_bytes":4685824,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":18626,"num_write":3,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,12,11,9,13,7,1],"acting":[8,12,11,9,13,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.18","version":"105'20","reported_seq":270,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-15T03:42:12.040856+0000","last_change":"2024-09-15T03:42:07.072210+0000","last_active":"2024-09-15T03:42:12.040856+0000","last_peered":"2024-09-15T03:42:12.040856+0000","last_clean":"2024-09-15T03:42:12.040856+0000","last_became_active":"2024-09-15T03:41:50.021784+0000","last_became_peered":"2024-09-15T03:41:50.021784+0000","last_unstale":"2024-09-15T03:42:12.040856+0000","last_undegraded":"2024-09-15T03:42:12.040856+0000","last_fullsized":"2024-09-15T03:42:12.040856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"43'9","last_scrub_stamp":"2024-09-15T03:41:17.762790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:17.762790+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:47.044535+0000","scrub_duration":0.107600868,"objects_trimmed":0,"snaptrim_duration":0.0022445049999999999,"stat_sum":{"num_bytes":7340032,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":6720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3637248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,0,1,5,7,12],"acting":[13,2,0,1,5,7,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.17","version":"105'87","reported_seq":324,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-15T03:42:12.054166+0000","last_change":"2024-09-15T03:42:07.087678+0000","last_active":"2024-09-15T03:42:12.054166+0000","last_peered":"2024-09-15T03:42:12.054166+0000","last_clean":"2024-09-15T03:42:12.054166+0000","last_became_active":"2024-09-15T03:41:19.031644+0000","last_became_peered":"2024-09-15T03:41:19.031644+0000","last_unstale":"2024-09-15T03:42:12.054166+0000","last_undegraded":"2024-09-15T03:42:12.054166+0000","last_fullsized":"2024-09-15T03:42:12.054166+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'40","last_scrub_stamp":"2024-09-15T03:41:30.774099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.774099+0000","objects_scrubbed":3,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:49.879526+0000","scrub_duration":0.003155648,"objects_trimmed":0,"snaptrim_duration":0.012574444000000001,"stat_sum":{"num_bytes":8552448,"num_objects":8,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":13,"num_read_kb":10181,"num_write":39,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,10,9,7,1,13],"acting":[5,15,10,9,7,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.16","version":"39'12","reported_seq":220,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.012278+0000","last_change":"2024-09-15T03:42:07.071076+0000","last_active":"2024-09-15T03:42:10.012278+0000","last_peered":"2024-09-15T03:42:10.012278+0000","last_clean":"2024-09-15T03:42:10.012278+0000","last_became_active":"2024-09-15T03:41:50.016734+0000","last_became_peered":"2024-09-15T03:41:50.016734+0000","last_unstale":"2024-09-15T03:42:10.012278+0000","last_undegraded":"2024-09-15T03:42:10.012278+0000","last_fullsized":"2024-09-15T03:42:10.012278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'12","last_scrub_stamp":"2024-09-15T03:41:32.804081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:32.804081+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:40.140365+0000","scrub_duration":0.0022082680000000002,"objects_trimmed":0,"snaptrim_duration":0.00092564800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15,13,8,9,5,10],"acting":[11,15,13,8,9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.15","version":"105'54","reported_seq":250,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-15T03:42:12.042404+0000","last_change":"2024-09-15T03:42:07.084720+0000","last_active":"2024-09-15T03:42:12.042404+0000","last_peered":"2024-09-15T03:42:12.042404+0000","last_clean":"2024-09-15T03:42:12.042404+0000","last_became_active":"2024-09-15T03:41:19.032261+0000","last_became_peered":"2024-09-15T03:41:19.032261+0000","last_unstale":"2024-09-15T03:42:12.042404+0000","last_undegraded":"2024-09-15T03:42:12.042404+0000","last_fullsized":"2024-09-15T03:42:12.042404+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'17","last_scrub_stamp":"2024-09-15T03:41:28.504954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:28.504954+0000","objects_scrubbed":3,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:36.482590+0000","scrub_duration":0.0034390330000000002,"objects_trimmed":0,"snaptrim_duration":0.0095388279999999992,"stat_sum":{"num_bytes":6340608,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":11334,"num_write":17,"num_write_kb":11088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0,13,14,2,15],"acting":[9,6,0,13,14,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":214,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.012249+0000","last_change":"2024-09-15T03:42:07.071096+0000","last_active":"2024-09-15T03:42:10.012249+0000","last_peered":"2024-09-15T03:42:10.012249+0000","last_clean":"2024-09-15T03:42:10.012249+0000","last_became_active":"2024-09-15T03:41:47.141772+0000","last_became_peered":"2024-09-15T03:41:47.141772+0000","last_unstale":"2024-09-15T03:42:10.012249+0000","last_undegraded":"2024-09-15T03:42:10.012249+0000","last_fullsized":"2024-09-15T03:42:10.012249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:52.583540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:52.583540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.661100+0000","scrub_duration":0.0034973769999999999,"objects_trimmed":0,"snaptrim_duration":0.00094328900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,15,1,13,8,7],"acting":[11,9,15,1,13,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":99,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.017058+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:10.017058+0000","last_peered":"2024-09-15T03:42:10.017058+0000","last_clean":"2024-09-15T03:42:10.017058+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:10.017058+0000","last_undegraded":"2024-09-15T03:42:10.017058+0000","last_fullsized":"2024-09-15T03:42:10.017058+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"101'22","reported_seq":276,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-15T03:42:09.355769+0000","last_change":"2024-09-15T03:42:07.070405+0000","last_active":"2024-09-15T03:42:09.355769+0000","last_peered":"2024-09-15T03:42:09.355769+0000","last_clean":"2024-09-15T03:42:09.355769+0000","last_became_active":"2024-09-15T03:41:12.026344+0000","last_became_peered":"2024-09-15T03:41:12.026344+0000","last_unstale":"2024-09-15T03:42:09.355769+0000","last_undegraded":"2024-09-15T03:42:09.355769+0000","last_fullsized":"2024-09-15T03:42:09.355769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'12","last_scrub_stamp":"2024-09-15T03:41:30.551305+0000","last_deep_scrub":"39'12","last_deep_scrub_stamp":"2024-09-15T03:41:30.551305+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.551305+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:57.833030+0000","scrub_duration":0.0038834,"objects_trimmed":0,"snaptrim_duration":0.00064149599999999997,"stat_sum":{"num_bytes":5652480,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":9652,"num_write":9,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.025078+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:11.025078+0000","last_peered":"2024-09-15T03:42:11.025078+0000","last_clean":"2024-09-15T03:42:11.025078+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:11.025078+0000","last_undegraded":"2024-09-15T03:42:11.025078+0000","last_fullsized":"2024-09-15T03:42:11.025078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"94'42","reported_seq":264,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.018301+0000","last_change":"2024-09-15T03:42:07.071359+0000","last_active":"2024-09-15T03:42:11.018301+0000","last_peered":"2024-09-15T03:42:11.018301+0000","last_clean":"2024-09-15T03:42:11.018301+0000","last_became_active":"2024-09-15T03:41:50.016701+0000","last_became_peered":"2024-09-15T03:41:50.016701+0000","last_unstale":"2024-09-15T03:42:11.018301+0000","last_undegraded":"2024-09-15T03:42:11.018301+0000","last_fullsized":"2024-09-15T03:42:11.018301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'23","last_scrub_stamp":"2024-09-15T03:41:27.592167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:27.592167+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:51.134452+0000","scrub_duration":0.123318016,"objects_trimmed":0,"snaptrim_duration":0.0066503029999999998,"stat_sum":{"num_bytes":1736704,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,6,9,0,8,5,11],"acting":[14,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":108,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.024119+0000","last_change":"2024-09-15T03:42:07.145738+0000","last_active":"2024-09-15T03:42:11.024119+0000","last_peered":"2024-09-15T03:42:11.024119+0000","last_clean":"2024-09-15T03:42:11.024119+0000","last_became_active":"2024-09-15T03:41:53.148724+0000","last_became_peered":"2024-09-15T03:41:53.148724+0000","last_unstale":"2024-09-15T03:42:11.024119+0000","last_undegraded":"2024-09-15T03:42:11.024119+0000","last_fullsized":"2024-09-15T03:42:11.024119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:28.348770+0000","scrub_duration":0.0017459419999999999,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":256,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.024441+0000","last_change":"2024-09-15T03:42:07.065801+0000","last_active":"2024-09-15T03:42:11.024441+0000","last_peered":"2024-09-15T03:42:11.024441+0000","last_clean":"2024-09-15T03:42:11.024441+0000","last_became_active":"2024-09-15T03:41:47.145039+0000","last_became_peered":"2024-09-15T03:41:47.145039+0000","last_unstale":"2024-09-15T03:42:11.024441+0000","last_undegraded":"2024-09-15T03:42:11.024441+0000","last_fullsized":"2024-09-15T03:42:11.024441+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:39.849670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:39.849670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:45.081635+0000","scrub_duration":0.0023338590000000002,"objects_trimmed":0,"snaptrim_duration":0.00092721299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,5,1,0,14,13],"acting":[8,2,5,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-15T03:42:09.355622+0000","last_change":"2024-09-15T03:41:47.140044+0000","last_active":"2024-09-15T03:42:09.355622+0000","last_peered":"2024-09-15T03:42:09.355622+0000","last_clean":"2024-09-15T03:42:09.355622+0000","last_became_active":"2024-09-15T03:41:47.139701+0000","last_became_peered":"2024-09-15T03:41:47.139701+0000","last_unstale":"2024-09-15T03:42:09.355622+0000","last_undegraded":"2024-09-15T03:42:09.355622+0000","last_fullsized":"2024-09-15T03:42:09.355622+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"101'61","reported_seq":351,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.030220+0000","last_change":"2024-09-15T03:42:07.071310+0000","last_active":"2024-09-15T03:42:10.030220+0000","last_peered":"2024-09-15T03:42:10.030220+0000","last_clean":"2024-09-15T03:42:10.030220+0000","last_became_active":"2024-09-15T03:41:50.021802+0000","last_became_peered":"2024-09-15T03:41:50.021802+0000","last_unstale":"2024-09-15T03:42:10.030220+0000","last_undegraded":"2024-09-15T03:42:10.030220+0000","last_fullsized":"2024-09-15T03:42:10.030220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'15","last_scrub_stamp":"2024-09-15T03:41:15.701370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.701370+0000","objects_scrubbed":3,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:33.489446+0000","scrub_duration":0.0086709649999999992,"objects_trimmed":0,"snaptrim_duration":0.00070245799999999999,"stat_sum":{"num_bytes":4816896,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":23,"num_write_kb":7584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,5,13,7,0,6],"acting":[12,14,5,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.3","version":"100'19","reported_seq":325,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.017787+0000","last_change":"2024-09-15T03:42:07.071505+0000","last_active":"2024-09-15T03:42:11.017787+0000","last_peered":"2024-09-15T03:42:11.017787+0000","last_clean":"2024-09-15T03:42:11.017787+0000","last_became_active":"2024-09-15T03:41:50.021721+0000","last_became_peered":"2024-09-15T03:41:50.021721+0000","last_unstale":"2024-09-15T03:42:11.017787+0000","last_undegraded":"2024-09-15T03:42:11.017787+0000","last_fullsized":"2024-09-15T03:42:11.017787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'18","last_scrub_stamp":"2024-09-15T03:41:40.701317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:40.701317+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:07.556129+0000","scrub_duration":0.0022822810000000002,"objects_trimmed":0,"snaptrim_duration":0.001724777,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,9,15,2,5,6],"acting":[13,7,9,15,2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.2","version":"29'2","reported_seq":106,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.029989+0000","last_change":"2024-09-15T03:41:55.835706+0000","last_active":"2024-09-15T03:42:10.029989+0000","last_peered":"2024-09-15T03:42:10.029989+0000","last_clean":"2024-09-15T03:42:10.029989+0000","last_became_active":"2024-09-15T03:41:19.030870+0000","last_became_peered":"2024-09-15T03:41:19.030870+0000","last_unstale":"2024-09-15T03:42:10.029989+0000","last_undegraded":"2024-09-15T03:42:10.029989+0000","last_fullsized":"2024-09-15T03:42:10.029989+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:07.326078+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,5],"acting":[12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.0","version":"100'44","reported_seq":338,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.018308+0000","last_change":"2024-09-15T03:42:07.070833+0000","last_active":"2024-09-15T03:42:11.018308+0000","last_peered":"2024-09-15T03:42:11.018308+0000","last_clean":"2024-09-15T03:42:11.018308+0000","last_became_active":"2024-09-15T03:41:50.021213+0000","last_became_peered":"2024-09-15T03:41:50.021213+0000","last_unstale":"2024-09-15T03:42:11.018308+0000","last_undegraded":"2024-09-15T03:42:11.018308+0000","last_fullsized":"2024-09-15T03:42:11.018308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'14","last_scrub_stamp":"2024-09-15T03:41:15.716351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.716351+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:22.481352+0000","scrub_duration":0.0021503820000000002,"objects_trimmed":0,"snaptrim_duration":0.0063155429999999999,"stat_sum":{"num_bytes":4227072,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":16,"num_read_kb":1157,"num_write":11,"num_write_kb":3984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2277376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,15,1,5,6,8],"acting":[14,2,15,1,5,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":100,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.017764+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:11.017764+0000","last_peered":"2024-09-15T03:42:11.017764+0000","last_clean":"2024-09-15T03:42:11.017764+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:11.017764+0000","last_undegraded":"2024-09-15T03:42:11.017764+0000","last_fullsized":"2024-09-15T03:42:11.017764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"95'24","reported_seq":268,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.024334+0000","last_change":"2024-09-15T03:42:07.065883+0000","last_active":"2024-09-15T03:42:11.024334+0000","last_peered":"2024-09-15T03:42:11.024334+0000","last_clean":"2024-09-15T03:42:11.024334+0000","last_became_active":"2024-09-15T03:41:50.026461+0000","last_became_peered":"2024-09-15T03:41:50.026461+0000","last_unstale":"2024-09-15T03:42:11.024334+0000","last_undegraded":"2024-09-15T03:42:11.024334+0000","last_fullsized":"2024-09-15T03:42:11.024334+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'9","last_scrub_stamp":"2024-09-15T03:41:20.521896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:20.521896+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:39.139808+0000","scrub_duration":0.0032032549999999999,"objects_trimmed":0,"snaptrim_duration":0.0010649539999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":721,"num_write":22,"num_write_kb":240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,9,5,10,2,13],"acting":[8,11,9,5,10,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.024301+0000","last_change":"2024-09-15T03:42:05.071989+0000","last_active":"2024-09-15T03:42:11.024301+0000","last_peered":"2024-09-15T03:42:11.024301+0000","last_clean":"2024-09-15T03:42:11.024301+0000","last_became_active":"2024-09-15T03:41:53.909931+0000","last_became_peered":"2024-09-15T03:41:53.909931+0000","last_unstale":"2024-09-15T03:42:11.024301+0000","last_undegraded":"2024-09-15T03:42:11.024301+0000","last_fullsized":"2024-09-15T03:42:11.024301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.252934+0000","scrub_duration":0.0014713300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"35'2","reported_seq":274,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.017790+0000","last_change":"2024-09-15T03:42:07.072798+0000","last_active":"2024-09-15T03:42:11.017790+0000","last_peered":"2024-09-15T03:42:11.017790+0000","last_clean":"2024-09-15T03:42:11.017790+0000","last_became_active":"2024-09-15T03:41:19.036787+0000","last_became_peered":"2024-09-15T03:41:19.036787+0000","last_unstale":"2024-09-15T03:42:11.017790+0000","last_undegraded":"2024-09-15T03:42:11.017790+0000","last_fullsized":"2024-09-15T03:42:11.017790+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'2","last_scrub_stamp":"2024-09-15T03:41:44.574995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:44.574995+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:48.638764+0000","scrub_duration":0.82793730399999999,"objects_trimmed":0,"snaptrim_duration":0.0027774520000000001,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,7,6,15,1,11],"acting":[13,2,7,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":91,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.020851+0000","last_change":"2024-09-15T03:40:50.350634+0000","last_active":"2024-09-15T03:42:11.020851+0000","last_peered":"2024-09-15T03:42:11.020851+0000","last_clean":"2024-09-15T03:42:11.020851+0000","last_became_active":"2024-09-15T03:40:46.660299+0000","last_became_peered":"2024-09-15T03:40:46.660299+0000","last_unstale":"2024-09-15T03:42:11.020851+0000","last_undegraded":"2024-09-15T03:42:11.020851+0000","last_fullsized":"2024-09-15T03:42:11.020851+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:40:45.641258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:15.557596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":234,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.017819+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:11.017819+0000","last_peered":"2024-09-15T03:42:11.017819+0000","last_clean":"2024-09-15T03:42:11.017819+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:11.017819+0000","last_undegraded":"2024-09-15T03:42:11.017819+0000","last_fullsized":"2024-09-15T03:42:11.017819+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"103'90","reported_seq":349,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.503397+0000","last_change":"2024-09-15T03:42:07.084671+0000","last_active":"2024-09-15T03:42:11.503397+0000","last_peered":"2024-09-15T03:42:11.503397+0000","last_clean":"2024-09-15T03:42:11.503397+0000","last_became_active":"2024-09-15T03:41:12.029274+0000","last_became_peered":"2024-09-15T03:41:12.029274+0000","last_unstale":"2024-09-15T03:42:11.503397+0000","last_undegraded":"2024-09-15T03:42:11.503397+0000","last_fullsized":"2024-09-15T03:42:11.503397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'13","last_scrub_stamp":"2024-09-15T03:41:15.740279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:15.740279+0000","objects_scrubbed":4,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:38.870215+0000","scrub_duration":0.0036761319999999999,"objects_trimmed":0,"snaptrim_duration":0.012832509000000001,"stat_sum":{"num_bytes":7061504,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":34,"num_read_kb":32374,"num_write":102,"num_write_kb":9936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2,1,14,15,6],"acting":[5,0,2,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.9","version":"35'7","reported_seq":277,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.018415+0000","last_change":"2024-09-15T03:42:07.065065+0000","last_active":"2024-09-15T03:42:11.018415+0000","last_peered":"2024-09-15T03:42:11.018415+0000","last_clean":"2024-09-15T03:42:11.018415+0000","last_became_active":"2024-09-15T03:41:19.035847+0000","last_became_peered":"2024-09-15T03:41:19.035847+0000","last_unstale":"2024-09-15T03:42:11.018415+0000","last_undegraded":"2024-09-15T03:42:11.018415+0000","last_fullsized":"2024-09-15T03:42:11.018415+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'7","last_scrub_stamp":"2024-09-15T03:41:30.900331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:30.900331+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:41.518622+0000","scrub_duration":0.0023468389999999999,"objects_trimmed":0,"snaptrim_duration":0.00050368400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,12,15,8,10],"acting":[14,1,0,12,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.a","version":"101'94","reported_seq":325,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-15T03:42:12.067356+0000","last_change":"2024-09-15T03:42:07.075710+0000","last_active":"2024-09-15T03:42:12.067356+0000","last_peered":"2024-09-15T03:42:12.067356+0000","last_clean":"2024-09-15T03:42:12.067356+0000","last_became_active":"2024-09-15T03:41:19.034361+0000","last_became_peered":"2024-09-15T03:41:19.034361+0000","last_unstale":"2024-09-15T03:42:12.067356+0000","last_undegraded":"2024-09-15T03:42:12.067356+0000","last_fullsized":"2024-09-15T03:42:12.067356+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010695812000000001,"stat_sum":{"num_bytes":12189696,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":26,"num_read_kb":5628,"num_write":52,"num_write_kb":14880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,1,9,14,6,11],"acting":[8,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.b","version":"100'42","reported_seq":352,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.028447+0000","last_change":"2024-09-15T03:42:07.083548+0000","last_active":"2024-09-15T03:42:11.028447+0000","last_peered":"2024-09-15T03:42:11.028447+0000","last_clean":"2024-09-15T03:42:11.028447+0000","last_became_active":"2024-09-15T03:41:50.018104+0000","last_became_peered":"2024-09-15T03:41:50.018104+0000","last_unstale":"2024-09-15T03:42:11.028447+0000","last_undegraded":"2024-09-15T03:42:11.028447+0000","last_fullsized":"2024-09-15T03:42:11.028447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'40","last_scrub_stamp":"2024-09-15T03:42:06.615211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:42:06.615211+0000","objects_scrubbed":4,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.541627+0000","scrub_duration":0.0033695489999999999,"objects_trimmed":0,"snaptrim_duration":0.018796686,"stat_sum":{"num_bytes":4653056,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":12296,"num_write":40,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":9863168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5,1,12,6,7,9],"acting":[14,5,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.c","version":"101'60","reported_seq":307,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.018430+0000","last_change":"2024-09-15T03:42:07.071701+0000","last_active":"2024-09-15T03:42:11.018430+0000","last_peered":"2024-09-15T03:42:11.018430+0000","last_clean":"2024-09-15T03:42:11.018430+0000","last_became_active":"2024-09-15T03:41:39.015749+0000","last_became_peered":"2024-09-15T03:41:39.015749+0000","last_unstale":"2024-09-15T03:42:11.018430+0000","last_undegraded":"2024-09-15T03:42:11.018430+0000","last_fullsized":"2024-09-15T03:42:11.018430+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070989579999999998,"stat_sum":{"num_bytes":2260992,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":14509,"num_write":49,"num_write_kb":7024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,0,10,6,12,5],"acting":[14,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.d","version":"101'46","reported_seq":342,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.017460+0000","last_change":"2024-09-15T03:42:07.064357+0000","last_active":"2024-09-15T03:42:10.017460+0000","last_peered":"2024-09-15T03:42:10.017460+0000","last_clean":"2024-09-15T03:42:10.017460+0000","last_became_active":"2024-09-15T03:41:50.021249+0000","last_became_peered":"2024-09-15T03:41:50.021249+0000","last_unstale":"2024-09-15T03:42:10.017460+0000","last_undegraded":"2024-09-15T03:42:10.017460+0000","last_fullsized":"2024-09-15T03:42:10.017460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000340749,"stat_sum":{"num_bytes":1982464,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":10422,"num_write":26,"num_write_kb":7200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,14,10,9,8,2],"acting":[7,0,14,10,9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.e","version":"104'92","reported_seq":391,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.026207+0000","last_change":"2024-09-15T03:42:07.360129+0000","last_active":"2024-09-15T03:42:11.026207+0000","last_peered":"2024-09-15T03:42:11.026207+0000","last_clean":"2024-09-15T03:42:11.026207+0000","last_became_active":"2024-09-15T03:41:39.015840+0000","last_became_peered":"2024-09-15T03:41:39.015840+0000","last_unstale":"2024-09-15T03:42:11.026207+0000","last_undegraded":"2024-09-15T03:42:11.026207+0000","last_fullsized":"2024-09-15T03:42:11.026207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'87","last_scrub_stamp":"2024-09-15T03:42:07.360044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.360044+0000","objects_scrubbed":3,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:11.109168+0000","scrub_duration":0.0027624279999999999,"objects_trimmed":0,"snaptrim_duration":0.00056164799999999999,"stat_sum":{"num_bytes":6062080,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":59,"num_read_kb":26752,"num_write":32,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,1,11,8,6,12],"acting":[2,10,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.f","version":"100'91","reported_seq":306,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.012351+0000","last_change":"2024-09-15T03:42:07.755815+0000","last_active":"2024-09-15T03:42:10.012351+0000","last_peered":"2024-09-15T03:42:10.012351+0000","last_clean":"2024-09-15T03:42:10.012351+0000","last_became_active":"2024-09-15T03:40:59.584644+0000","last_became_peered":"2024-09-15T03:40:59.584644+0000","last_unstale":"2024-09-15T03:42:10.012351+0000","last_undegraded":"2024-09-15T03:42:10.012351+0000","last_fullsized":"2024-09-15T03:42:10.012351+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:40:58.956298+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.68537477499999999,"stat_sum":{"num_bytes":12206080,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":13499,"num_write":98,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,14,13,8,10,0],"acting":[11,2,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.10","version":"95'69","reported_seq":296,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.017650+0000","last_change":"2024-09-15T03:42:07.074650+0000","last_active":"2024-09-15T03:42:11.017650+0000","last_peered":"2024-09-15T03:42:11.017650+0000","last_clean":"2024-09-15T03:42:11.017650+0000","last_became_active":"2024-09-15T03:41:50.019077+0000","last_became_peered":"2024-09-15T03:41:50.019077+0000","last_unstale":"2024-09-15T03:42:11.017650+0000","last_undegraded":"2024-09-15T03:42:11.017650+0000","last_fullsized":"2024-09-15T03:42:11.017650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"58'33","last_scrub_stamp":"2024-09-15T03:41:29.106893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:29.106893+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:49.054548+0000","scrub_duration":0.39459468800000003,"objects_trimmed":0,"snaptrim_duration":0.0048221890000000002,"stat_sum":{"num_bytes":4423680,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":579,"num_write":12,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4079616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,5,8,0,11],"acting":[13,6,7,5,8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.11","version":"103'33","reported_seq":283,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.024301+0000","last_change":"2024-09-15T03:42:07.070028+0000","last_active":"2024-09-15T03:42:11.024301+0000","last_peered":"2024-09-15T03:42:11.024301+0000","last_clean":"2024-09-15T03:42:11.024301+0000","last_became_active":"2024-09-15T03:41:17.030558+0000","last_became_peered":"2024-09-15T03:41:17.030558+0000","last_unstale":"2024-09-15T03:42:11.024301+0000","last_undegraded":"2024-09-15T03:42:11.024301+0000","last_fullsized":"2024-09-15T03:42:11.024301+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'17","last_scrub_stamp":"2024-09-15T03:41:17.535319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:17.535319+0000","objects_scrubbed":3,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:20.364025+0000","scrub_duration":0.003091599,"objects_trimmed":0,"snaptrim_duration":0.00056524600000000004,"stat_sum":{"num_bytes":6602752,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":5188,"num_write":8,"num_write_kb":5264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,15,9,5,6,1],"acting":[10,11,15,9,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.12","version":"35'2","reported_seq":224,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-15T03:42:11.020209+0000","last_change":"2024-09-15T03:42:07.075474+0000","last_active":"2024-09-15T03:42:11.020209+0000","last_peered":"2024-09-15T03:42:11.020209+0000","last_clean":"2024-09-15T03:42:11.020209+0000","last_became_active":"2024-09-15T03:41:50.026297+0000","last_became_peered":"2024-09-15T03:41:50.026297+0000","last_unstale":"2024-09-15T03:42:11.020209+0000","last_undegraded":"2024-09-15T03:42:11.020209+0000","last_fullsized":"2024-09-15T03:42:11.020209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'2","last_scrub_stamp":"2024-09-15T03:41:29.466491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:29.466491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:36.023400+0000","scrub_duration":0.0021447580000000001,"objects_trimmed":0,"snaptrim_duration":0.000861165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8,6,11,2,15],"acting":[9,7,8,6,11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"3.13","version":"103'66","reported_seq":396,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-15T03:42:10.022190+0000","last_change":"2024-09-15T03:42:07.071679+0000","last_active":"2024-09-15T03:42:10.022190+0000","last_peered":"2024-09-15T03:42:10.022190+0000","last_clean":"2024-09-15T03:42:10.022190+0000","last_became_active":"2024-09-15T03:41:50.024851+0000","last_became_peered":"2024-09-15T03:41:50.024851+0000","last_unstale":"2024-09-15T03:42:10.022190+0000","last_undegraded":"2024-09-15T03:42:10.022190+0000","last_fullsized":"2024-09-15T03:42:10.022190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'21","last_scrub_stamp":"2024-09-15T03:41:19.809341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:58.956298+0000","last_clean_scrub_stamp":"2024-09-15T03:41:19.809341+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:21.113176+0000","scrub_duration":0.0033873140000000002,"objects_trimmed":0,"snaptrim_duration":0.001534702,"stat_sum":{"num_bytes":3227648,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,5,2,0,15,10],"acting":[11,7,5,2,0,15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3d","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":104546304,"num_objects":71,"num_object_clones":28,"num_object_copies":497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":5,"num_read":311,"num_read_kb":172926,"num_write":579,"num_write_kb":125040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":53411840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185503744,"data_stored":262684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":0},"log_size":1108,"ondisk_log_size":1108,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315516,"kb_used_data":14000,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056324,"statfs":{"total":96636764160,"available":96313675776,"internally_reserved":0,"allocated":14336000,"data_stored":19572296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":8.6400000000000006,"15min":8.6400000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":170.74299999999999,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":4,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6849999999999996,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6980000000000004,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.39700000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.372},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.34999999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":0.505},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.67600000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":23.472999999999999,"5min":23.472999999999999,"15min":23.472999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":478.76900000000001,"5min":478.76900000000001,"15min":478.76900000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.53200000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.49199999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317044,"kb_used_data":15520,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94054796,"statfs":{"total":96636764160,"available":96312111104,"internally_reserved":0,"allocated":15892480,"data_stored":19188605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.79600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":48.625999999999998,"5min":48.625999999999998,"15min":48.625999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":913.03200000000004,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.501}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.872},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.755}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.82699999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":13,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":30.189,"5min":30.189,"15min":30.189},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":560.85299999999995,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.92400000000000004}]},{"osd":15,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.88600000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280532,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315492,"kb_used_data":13968,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056348,"statfs":{"total":96636764160,"available":96313700352,"internally_reserved":0,"allocated":14303232,"data_stored":19511541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.72999999999999998},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.85099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":18.027999999999999,"5min":18.027999999999999,"15min":18.027999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":379.39100000000002,"5min":379.39100000000002,"15min":379.39100000000002},"last":1.0980000000000001},{"interface":"front","average":{"1min":41.061999999999998,"5min":41.061999999999998,"15min":41.061999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":886.51800000000003,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.94299999999999995}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.89400000000000002},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.91900000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1739999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":13.311,"5min":13.311,"15min":13.311},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":274.65100000000001,"5min":274.65100000000001,"15min":274.65100000000001},"last":1.0660000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":1.048},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":1.0029999999999999}]}]},{"osd":0,"up_from":21,"seq":90194313237,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315124,"kb_used_data":13604,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056716,"statfs":{"total":96636764160,"available":96314077184,"internally_reserved":0,"allocated":13930496,"data_stored":18755720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":41.966000000000001,"5min":41.966000000000001,"15min":41.966000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":703.98400000000004,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.745}]},{"osd":7,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":25.172999999999998,"5min":25.172999999999998,"15min":25.172999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":419.01499999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.68200000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.39800000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70199999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":3.972,"5min":3.972,"15min":3.972},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":62.659999999999997,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.80100000000000005}]}]},{"osd":6,"up_from":21,"seq":90194313236,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317880,"kb_used_data":16356,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94053960,"statfs":{"total":96636764160,"available":96311255040,"internally_reserved":0,"allocated":16748544,"data_stored":20720509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":37.668999999999997,"5min":37.668999999999997,"15min":37.668999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":668.923,"5min":668.923,"15min":668.923},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.70799999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51300000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":15.977,"5min":15.977,"15min":15.977},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":290.32400000000001,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.61799999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.502}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.73099999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":15,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.85599999999999998}]}]},{"osd":15,"up_from":21,"seq":90194313238,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312844,"kb_used_data":11324,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94058996,"statfs":{"total":96636764160,"available":96316411904,"internally_reserved":0,"allocated":11595776,"data_stored":18694280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,4,5,6,7,8,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":45.473999999999997,"5min":45.473999999999997,"15min":45.473999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":764.62199999999996,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.47399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41399999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46500000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":18.594999999999999,"5min":18.594999999999999,"15min":18.594999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":305.18200000000002,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.52200000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":58.883000000000003,"5min":58.883000000000003,"15min":58.883000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":782.05200000000002,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.377}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40500000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.438},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.497}]},{"osd":13,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56599999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.61899999999999999}]}]},{"osd":3,"up_from":0,"seq":292057776137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":22,"seq":94489280532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315836,"kb_used_data":14316,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056004,"statfs":{"total":96636764160,"available":96313348096,"internally_reserved":0,"allocated":14659584,"data_stored":19161802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60099999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.68000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.54600000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":12,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.626}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":40.006999999999998,"5min":40.006999999999998,"15min":40.006999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":740.69799999999998,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.76300000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.68600000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315644,"kb_used_data":14120,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056196,"statfs":{"total":96636764160,"available":96313544704,"internally_reserved":0,"allocated":14458880,"data_stored":15645565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.372},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67100000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.177,"5min":3.177,"15min":3.177},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.81599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79300000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.94399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":33.399999999999999,"5min":33.399999999999999,"15min":33.399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":557.40099999999995,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.73399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.80700000000000005}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":41.654000000000003,"5min":41.654000000000003,"15min":41.654000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":694.31399999999996,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.749}]},{"osd":13,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":11.574,"5min":11.574,"15min":11.574},"last":0.93300000000000005}]},{"osd":14,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":1.0029999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]}]},{"osd":12,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306164,"kb_used_data":8804,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065676,"statfs":{"total":96636764160,"available":96323252224,"internally_reserved":0,"allocated":9015296,"data_stored":10178715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.34999999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":10.916,"5min":10.916,"15min":10.916},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":201.73599999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.44600000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":14.640000000000001,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.46600000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.497},{"interface":"front","average":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":818.68700000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.51700000000000002}]},{"osd":15,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":34.192,"5min":34.192,"15min":34.192},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":703.52499999999998,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.47499999999999998}]}]},{"osd":1,"up_from":22,"seq":94489280532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318172,"kb_used_data":16648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94053668,"statfs":{"total":96636764160,"available":96310956032,"internally_reserved":0,"allocated":17047552,"data_stored":22763835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.46899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.63},{"interface":"front","average":{"1min":10.869999999999999,"5min":10.869999999999999,"15min":10.869999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":214.91,"5min":214.91,"15min":214.91},"last":0.66300000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.621},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.61099999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.50600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.69999999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":4,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":22,"seq":94489280532,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316468,"kb_used_data":14952,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94055372,"statfs":{"total":96636764160,"available":96312700928,"internally_reserved":0,"allocated":15310848,"data_stored":22372857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.71099999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.67300000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70199999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.629},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.64000000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.75},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.77800000000000002}]}]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315192,"kb_used_data":13676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056648,"statfs":{"total":96636764160,"available":96314007552,"internally_reserved":0,"allocated":14004224,"data_stored":15946792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":405.41500000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.308}]},{"osd":1,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.625},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69499999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.42999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.49099999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":22.788,"5min":22.788,"15min":22.788},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":398.71699999999998,"5min":398.71699999999998,"15min":398.71699999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":51.093000000000004,"5min":51.093000000000004,"15min":51.093000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":908.44200000000001,"5min":908.44200000000001,"15min":908.44200000000001},"last":0.72999999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":8.952,"5min":8.952,"15min":8.952},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":149.09,"5min":149.09,"15min":149.09},"last":0.61199999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":13,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.68200000000000005}]},{"osd":14,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51700000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280534,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315832,"kb_used_data":14316,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056008,"statfs":{"total":96636764160,"available":96313352192,"internally_reserved":0,"allocated":14659584,"data_stored":20239400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":10.802,"5min":10.802,"15min":10.802},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":151.80600000000001,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.78900000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":7.327,"5min":7.327,"15min":7.327},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":98.858000000000004,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.877}]},{"osd":11,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":37.173999999999999,"5min":37.173999999999999,"15min":37.173999999999999},"last":337.03300000000002},{"interface":"front","average":{"1min":14.319000000000001,"5min":14.319000000000001,"15min":14.319000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":203.91399999999999,"5min":203.91399999999999,"15min":203.91399999999999},"last":0.69399999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.85599999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.80400000000000005},{"interface":"front","average":{"1min":5.7759999999999998,"5min":5.7759999999999998,"15min":5.7759999999999998},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":73.846999999999994,"5min":73.846999999999994,"15min":73.846999999999994},"last":0.79700000000000004}]},{"osd":15,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.81399999999999995}]}]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308864,"kb_used_data":11508,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94062976,"statfs":{"total":96636764160,"available":96320487424,"internally_reserved":0,"allocated":11784192,"data_stored":18076712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.40000000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":9,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56799999999999995}]},{"osd":11,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.64700000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.68899999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:41:46 2024","interfaces":[{"interface":"back","average":{"1min":16.597000000000001,"5min":16.597000000000001,"15min":16.597000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":287.86900000000003,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.70299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12406784,"data_stored":17850368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16887808,"data_stored":24117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12242944,"data_stored":18116608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6684672,"data_stored":7659520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13750272,"data_stored":21446656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15204352,"data_stored":19804160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13131776,"data_stored":18255872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12443648,"data_stored":15020032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13099008,"data_stored":19312640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":10223616,"data_stored":17149952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12914688,"data_stored":14729216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":7487488,"data_stored":10145792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13660160,"data_stored":21008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14393344,"data_stored":19378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10526720,"data_stored":18243584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:14.692 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-15T03:42:14.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-15T03:42:14.738 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-15T03:42:14.738+0000 7f9cab3f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33250) UID: 0 2024-09-15T03:42:14.781 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.781 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.789 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-15T03:42:14.806 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-15T03:42:14.839 INFO:tasks.ceph.osd.14.smithi078.stderr:2024-09-15T03:42:14.838+0000 7fbb3a7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32807) UID: 0 2024-09-15T03:42:14.888 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.888 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:14.896 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:42:14.939 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-15T03:42:14.939 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-15T03:42:14.940 INFO:tasks.ceph.osd.7.smithi105.stderr:2024-09-15T03:42:14.939+0000 7f54745f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33104) UID: 0 2024-09-15T03:42:14.961 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-15T03:42:14.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-15T03:42:14.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-15T03:42:14.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-15T03:42:14.993 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-09-15T03:42:14.994 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-15T03:42:15.000 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-15T03:42:15.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-15T03:42:15.002 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-15T03:42:15.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-15T03:42:15.020 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-15T03:42:15.041 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-15T03:42:15.041 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-15T03:42:15.043 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-09-15T03:42:15.042+0000 7fd243bf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43936) UID: 0 2024-09-15T03:42:15.061 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-15T03:42:15.062 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-15T03:42:15.081 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-15T03:42:15.082 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-15T03:42:15.090 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_0' removed 2024-09-15T03:42:15.090 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:42:15.106 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-15T03:42:15.115 INFO:tasks.thrashosds:joining thrashosds 2024-09-15T03:42:15.116 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-15T03:42:15.117 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-15T03:42:15.126 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-15T03:42:15.127 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-15T03:42:15.139 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:15.139 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-09-15T03:42:15.139 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-09-15T03:42:15.139 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-09-15T03:42:15.139 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:15.146 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-15T03:42:15.147 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-15T03:42:15.148 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-15T03:42:15.166 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-15T03:42:15.166 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-15T03:42:15.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-15T03:42:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-15T03:42:15.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:15.206+0000 7fcb2259c640 0 -- 172.21.15.5:0/980005164 >> [v2:172.21.15.64:6823/33261,v1:172.21.15.64:6825/33261] conn(0x7fcb24190380 msgr2=0x7fcb241927a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-15T03:42:15.213 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-15T03:42:15.233 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-15T03:42:15.234 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-15T03:42:15.246 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-09-15T03:42:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-09-15T03:42:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-09-15T03:42:15.246 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-09-15T03:42:15.247 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-09-15T03:42:15.254 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-15T03:42:15.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-15T03:42:17.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-15T03:42:18.097 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-15T03:42:18.097 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-15T03:42:18.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-09-15T03:42:18.106 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-15T03:42:18.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-15T03:42:18.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-15T03:42:18.399+0000 7f8f507f1640 -1 mon.a@0(leader).osd e109 definitely_dead 0 2024-09-15T03:42:19.113 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.3. 2024-09-15T03:42:19.114 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-15T03:42:19.130 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-15T03:42:19.130 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-15T03:42:19.130 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-15T03:42:19.130 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-09-15T03:42:19.466 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-15T03:42:19.466+0000 7f8f507f1640 -1 mon.a@0(leader).osd e110 definitely_dead 0 2024-09-15T03:42:20.121 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.4. 2024-09-15T03:42:20.138 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-15T03:42:20.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-15T03:42:20.265 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a: {} 2024-09-15T03:42:20.265 INFO:teuthology.orchestra.run.smithi005.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:42:20.278 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b: {} 2024-09-15T03:42:20.278 INFO:teuthology.orchestra.run.smithi005.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:42:20.292 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c: {} 2024-09-15T03:42:20.292 INFO:teuthology.orchestra.run.smithi005.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-15T03:42:20.302 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-15T03:42:21.123 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-09-15T03:42:21.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-15T03:42:22.129 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-15T03:42:22.145 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-15T03:42:23.134 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-09-15T03:42:23.151 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-15T03:42:24.139 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-15T03:42:24.156 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-15T03:42:25.151 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-09-15T03:42:25.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-15T03:42:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-15T03:42:26.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-15T03:42:27.167 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.12 to 1 (10000) 2024-09-15T03:42:27.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-15T03:42:28.172 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-15T03:42:28.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-15T03:42:29.178 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-09-15T03:42:29.194 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-15T03:42:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-15T03:42:30.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-15T03:42:31.191 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-09-15T03:42:31.207 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-15T03:42:32.196 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-15T03:42:32.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-15T03:42:33.201 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-09-15T03:42:33.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-15T03:42:34.212 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-15T03:42:34.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-15T03:42:35.217 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.13 to 1 (10000) 2024-09-15T03:42:35.236 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-15T03:42:36.225 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-15T03:42:36.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-15T03:42:37.231 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-09-15T03:42:37.247 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-15T03:42:38.236 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-15T03:42:38.252 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-15T03:42:39.240 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-09-15T03:42:39.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-15T03:42:40.248 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-15T03:42:40.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-15T03:42:41.255 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-09-15T03:42:41.272 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-15T03:42:42.261 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-15T03:42:42.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-15T03:42:43.266 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.14 to 1 (10000) 2024-09-15T03:42:43.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-15T03:42:44.274 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-15T03:42:44.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-15T03:42:45.284 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-09-15T03:42:45.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-15T03:42:46.293 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-15T03:42:46.310 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-15T03:42:47.298 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-09-15T03:42:47.313 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-15T03:42:48.303 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-15T03:42:48.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-15T03:42:49.308 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.15 to 1 (10000) 2024-09-15T03:42:49.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-15T03:42:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-15T03:42:50.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-15T03:42:51.330 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-09-15T03:42:51.347 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-15T03:42:52.338 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-15T03:42:52.355 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-15T03:42:52.355 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-15T03:42:52.355 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-15T03:42:52.355 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-15T03:42:52.355 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:42:52.355 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:42:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:52.641 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":143,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:42:52.334021+0000","last_up_change":"2024-09-15T03:41:35.013591+0000","last_in_change":"2024-09-15T03:42:20.111918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":46,"up_from":68,"up_thru":113,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43941},{"type":"v1","addr":"172.21.15.105:6803","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43941},{"type":"v1","addr":"172.21.15.105:6805","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43941},{"type":"v1","addr":"172.21.15.105:6809","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43941},{"type":"v1","addr":"172.21.15.105:6807","nonce":43941}]},"public_addr":"172.21.15.105:6803/43941","cluster_addr":"172.21.15.105:6805/43941","heartbeat_back_addr":"172.21.15.105:6809/43941","heartbeat_front_addr":"172.21.15.105:6807/43941","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:49.977726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.988005+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:46.256258+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.516008+0000","dead_epoch":47},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:09.575529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:43.785792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:04.563391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.889587+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:52.239895+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:02.352976+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:51.140062+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:41.474463+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:55.629737+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:40.572229+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.006408+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:51.897727+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:42:52.652 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:42:52.652 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:42:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:52.935 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":143,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:42:52.334021+0000","last_up_change":"2024-09-15T03:41:35.013591+0000","last_in_change":"2024-09-15T03:42:20.111918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":46,"up_from":68,"up_thru":113,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43941},{"type":"v1","addr":"172.21.15.105:6803","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43941},{"type":"v1","addr":"172.21.15.105:6805","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43941},{"type":"v1","addr":"172.21.15.105:6809","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43941},{"type":"v1","addr":"172.21.15.105:6807","nonce":43941}]},"public_addr":"172.21.15.105:6803/43941","cluster_addr":"172.21.15.105:6805/43941","heartbeat_back_addr":"172.21.15.105:6809/43941","heartbeat_front_addr":"172.21.15.105:6807/43941","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:49.977726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.988005+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:46.256258+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.516008+0000","dead_epoch":47},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:09.575529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:43.785792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:04.563391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.889587+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:52.239895+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:02.352976+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:51.140062+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:41.474463+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:55.629737+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:40.572229+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.006408+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:51.897727+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:42:52.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:42:52.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:42:52.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:42:52.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:42:52.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:42:52.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:42:52.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-15T03:42:52.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-15T03:42:52.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-15T03:42:52.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-15T03:42:52.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-15T03:42:52.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-15T03:42:52.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-15T03:42:52.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-15T03:42:52.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-15T03:42:52.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-15T03:42:53.319 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:53.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:42:53.369 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:42:53.394 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:42:53.410 INFO:teuthology.orchestra.run.smithi005.stdout:90194313246 2024-09-15T03:42:53.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:42:53.413 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-15T03:42:53.417 INFO:teuthology.orchestra.run.smithi005.stdout:292057776146 2024-09-15T03:42:53.418 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.418 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-15T03:42:53.419 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:42:53.434 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:53.434 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-15T03:42:53.439 INFO:teuthology.orchestra.run.smithi005.stdout:90194313245 2024-09-15T03:42:53.439 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.440 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-15T03:42:53.440 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:53.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-15T03:42:53.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-15T03:42:53.442 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.443 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-15T03:42:53.450 INFO:teuthology.orchestra.run.smithi005.stdout:90194313247 2024-09-15T03:42:53.450 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-15T03:42:53.461 INFO:teuthology.orchestra.run.smithi005.stdout:94489280543 2024-09-15T03:42:53.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-15T03:42:53.498 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:53.499 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:42:53.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-15T03:42:53.547+0000 7fbb32895640 0 --2- 172.21.15.5:0/356908387 >> v2:172.21.15.78:3300/0 conn(0x7fbb2c155360 0x7fbb2c0c1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:42:53.772 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:53.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-15T03:42:54.240 INFO:teuthology.orchestra.run.smithi005.stdout:94489280540 2024-09-15T03:42:54.258 INFO:teuthology.orchestra.run.smithi005.stdout:94489280539 2024-09-15T03:42:54.275 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280540 for osd.4 2024-09-15T03:42:54.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280539 for osd.2 2024-09-15T03:42:54.426 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:54.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.5 2024-09-15T03:42:54.455 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.472 INFO:teuthology.orchestra.run.smithi005.stdout:94489280543 2024-09-15T03:42:54.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280543 got 94489280543 for osd.9 2024-09-15T03:42:54.492 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.504 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:54.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.14 2024-09-15T03:42:54.539 INFO:teuthology.orchestra.run.smithi005.stdout:292057776146 2024-09-15T03:42:54.540 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:54.540 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.550 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:54.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.10 2024-09-15T03:42:54.571 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.578 INFO:teuthology.orchestra.run.smithi005.stdout:90194313246 2024-09-15T03:42:54.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 292057776146 got 292057776146 for osd.3 2024-09-15T03:42:54.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.12 2024-09-15T03:42:54.580 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.580 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.580 INFO:teuthology.orchestra.run.smithi005.stdout:90194313245 2024-09-15T03:42:54.580 INFO:teuthology.orchestra.run.smithi005.stdout:90194313247 2024-09-15T03:42:54.590 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:54.599 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:54.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.0 2024-09-15T03:42:54.602 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313245 for osd.6 2024-09-15T03:42:54.602 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:54.602 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313247 got 90194313247 for osd.15 2024-09-15T03:42:54.605 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.11 2024-09-15T03:42:54.612 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.1 2024-09-15T03:42:54.616 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.13 2024-09-15T03:42:54.617 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.633 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:54.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.8 2024-09-15T03:42:54.651 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:54.654 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:54.667 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.7 2024-09-15T03:42:54.667 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:55.276 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:42:55.278 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:42:55.571 INFO:teuthology.orchestra.run.smithi005.stdout:94489280541 2024-09-15T03:42:55.576 INFO:teuthology.orchestra.run.smithi005.stdout:94489280542 2024-09-15T03:42:55.584 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280541 for osd.2 2024-09-15T03:42:55.585 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:55.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280542 for osd.4 2024-09-15T03:42:55.589 DEBUG:teuthology.parallel:result is None 2024-09-15T03:42:55.589 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-15T03:42:55.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:55.923 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-15T03:42:55.356977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4851100,"kb_used_data":35048,"kb_used_omap":26,"kb_used_meta":4815781,"kb_avail":1505098340,"statfs":{"total":1546188226560,"available":1541220700160,"internally_reserved":0,"allocated":35889152,"data_stored":21592199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4931360464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.317917"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.332176+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:51.332176+0000","last_peered":"2024-09-15T03:42:51.332176+0000","last_clean":"2024-09-15T03:42:51.332176+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:51.332176+0000","last_undegraded":"2024-09-15T03:42:51.332176+0000","last_fullsized":"2024-09-15T03:42:51.332176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.329944+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:51.329944+0000","last_peered":"2024-09-15T03:42:51.329944+0000","last_clean":"2024-09-15T03:42:51.329944+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:51.329944+0000","last_undegraded":"2024-09-15T03:42:51.329944+0000","last_fullsized":"2024-09-15T03:42:51.329944+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657364+0000","last_change":"2024-09-15T03:42:41.277510+0000","last_active":"2024-09-15T03:42:50.657364+0000","last_peered":"2024-09-15T03:42:50.657364+0000","last_clean":"2024-09-15T03:42:50.657364+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:42:50.657364+0000","last_undegraded":"2024-09-15T03:42:50.657364+0000","last_fullsized":"2024-09-15T03:42:50.657364+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.091680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.864286+0000","last_change":"2024-09-15T03:42:21.252540+0000","last_active":"2024-09-15T03:42:50.864286+0000","last_peered":"2024-09-15T03:42:50.864286+0000","last_clean":"2024-09-15T03:42:50.864286+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:42:50.864286+0000","last_undegraded":"2024-09-15T03:42:50.864286+0000","last_fullsized":"2024-09-15T03:42:50.864286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:16.370806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:16.370806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:44.843373+0000","scrub_duration":0.0019016580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":153,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153580+0000","last_change":"2024-09-15T03:42:23.138835+0000","last_active":"2024-09-15T03:42:51.153580+0000","last_peered":"2024-09-15T03:42:51.153580+0000","last_clean":"2024-09-15T03:42:51.153580+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:42:51.153580+0000","last_undegraded":"2024-09-15T03:42:51.153580+0000","last_fullsized":"2024-09-15T03:42:51.153580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:04.333008+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657590+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:50.657590+0000","last_peered":"2024-09-15T03:42:50.657590+0000","last_clean":"2024-09-15T03:42:50.657590+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:50.657590+0000","last_undegraded":"2024-09-15T03:42:50.657590+0000","last_fullsized":"2024-09-15T03:42:50.657590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153579+0000","last_change":"2024-09-15T03:42:20.126436+0000","last_active":"2024-09-15T03:42:51.153579+0000","last_peered":"2024-09-15T03:42:51.153579+0000","last_clean":"2024-09-15T03:42:51.153579+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:42:51.153579+0000","last_undegraded":"2024-09-15T03:42:51.153579+0000","last_fullsized":"2024-09-15T03:42:51.153579+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:17.090770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":118,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.660207+0000","last_change":"2024-09-15T03:42:33.209224+0000","last_active":"2024-09-15T03:42:50.660207+0000","last_peered":"2024-09-15T03:42:50.660207+0000","last_clean":"2024-09-15T03:42:50.660207+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:42:50.660207+0000","last_undegraded":"2024-09-15T03:42:50.660207+0000","last_fullsized":"2024-09-15T03:42:50.660207+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T03:42:17.024578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:17.024578+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.982049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":269,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657603+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:50.657603+0000","last_peered":"2024-09-15T03:42:50.657603+0000","last_clean":"2024-09-15T03:42:50.657603+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:50.657603+0000","last_undegraded":"2024-09-15T03:42:50.657603+0000","last_fullsized":"2024-09-15T03:42:50.657603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304160,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067680,"statfs":{"total":96636764160,"available":96325304320,"internally_reserved":0,"allocated":2703360,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":1.202},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.96199999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":1.224},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.94699999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":1},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":1.0309999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.93899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.97899999999999998}]}]},{"osd":14,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.58199999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.75}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.626}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.83599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69999999999999996}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.81000000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.84599999999999997}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304164,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067676,"statfs":{"total":96636764160,"available":96325300224,"internally_reserved":0,"allocated":2711552,"data_stored":1816766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.504},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.624}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.88200000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.47199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.63400000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91300000000000003},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.84599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.73299999999999998},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.71099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.90000000000000002}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.94699999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303616,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068224,"statfs":{"total":96636764160,"available":96325861376,"internally_reserved":0,"allocated":2150400,"data_stored":1269454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.622},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.72599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.57599999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.0069999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.878}]}]},{"osd":6,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.88}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.625},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.81000000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.436},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.45900000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":54.079000000000001,"5min":54.079000000000001,"15min":54.079000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":909.03399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.52400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39400000000000002},{"interface":"front","average":{"1min":41.031999999999996,"5min":41.031999999999996,"15min":41.031999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":686.33799999999997,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.503}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73799999999999999}]}]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.72899999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48699999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.54800000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.61599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65600000000000003}]}]},{"osd":3,"up_from":68,"seq":292057776147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299556,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072284,"statfs":{"total":96636764160,"available":96330018816,"internally_reserved":0,"allocated":2187264,"data_stored":1288422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.73599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.625},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72099999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.622}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.501},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.442}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.53400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.42899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.40300000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.68999999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.60999999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.48399999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.80300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.501}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.68500000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":47.381,"5min":47.381,"15min":47.381},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":981.20500000000004,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53700000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89000000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.87},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755}]},{"osd":15,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.64100000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299468,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072372,"statfs":{"total":96636764160,"available":96330108928,"internally_reserved":0,"allocated":2154496,"data_stored":1269473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.84599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.61899999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.88600000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.97899999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.87}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.81899999999999995}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303624,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068216,"statfs":{"total":96636764160,"available":96325853184,"internally_reserved":0,"allocated":2150400,"data_stored":1269454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.78800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.54400000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.73499999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.51200000000000001},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.47999999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.58799999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.745}]}]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303632,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068208,"statfs":{"total":96636764160,"available":96325844992,"internally_reserved":0,"allocated":2154496,"data_stored":1270032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.42399999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.442}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.63100000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.502}]},{"osd":14,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":22.876000000000001,"5min":22.876000000000001,"15min":22.876000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":353.327,"5min":353.327,"15min":353.327},"last":0.57899999999999996},{"interface":"front","average":{"1min":61.741999999999997,"5min":61.741999999999997,"15min":61.741999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":974.78599999999994,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.61599999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66900000000000004}]}]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303648,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068192,"statfs":{"total":96636764160,"available":96325828608,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63700000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.752}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.41199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.78600000000000003}]},{"osd":13,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.623},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53800000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.57599999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.80700000000000005}]}]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82499999999999996},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":1.262}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":1.276},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96699999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.92800000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":1.2909999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":1.248}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.94899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.716},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.671}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.83999999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":1.232},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.98199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.70799999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.72999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":1},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":1.0640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":1.032},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":1.159}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.92800000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":1.1100000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":10.291,"5min":10.291,"15min":10.291},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":176.59299999999999,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":1.024},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.86099999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.748},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.82699999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.92300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69099999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91000000000000003},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.81100000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.079}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:55.989 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:56.272 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:56.284 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T03:42:56.040357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4851136,"kb_used_data":35080,"kb_used_omap":26,"kb_used_meta":4815781,"kb_avail":1505098304,"statfs":{"total":1546188226560,"available":1541220663296,"internally_reserved":0,"allocated":35921920,"data_stored":21611197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4931360464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.726700"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.332176+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:51.332176+0000","last_peered":"2024-09-15T03:42:51.332176+0000","last_clean":"2024-09-15T03:42:51.332176+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:51.332176+0000","last_undegraded":"2024-09-15T03:42:51.332176+0000","last_fullsized":"2024-09-15T03:42:51.332176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.329944+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:51.329944+0000","last_peered":"2024-09-15T03:42:51.329944+0000","last_clean":"2024-09-15T03:42:51.329944+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:51.329944+0000","last_undegraded":"2024-09-15T03:42:51.329944+0000","last_fullsized":"2024-09-15T03:42:51.329944+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657364+0000","last_change":"2024-09-15T03:42:41.277510+0000","last_active":"2024-09-15T03:42:50.657364+0000","last_peered":"2024-09-15T03:42:50.657364+0000","last_clean":"2024-09-15T03:42:50.657364+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:42:50.657364+0000","last_undegraded":"2024-09-15T03:42:50.657364+0000","last_fullsized":"2024-09-15T03:42:50.657364+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.091680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.864286+0000","last_change":"2024-09-15T03:42:21.252540+0000","last_active":"2024-09-15T03:42:50.864286+0000","last_peered":"2024-09-15T03:42:50.864286+0000","last_clean":"2024-09-15T03:42:50.864286+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:42:50.864286+0000","last_undegraded":"2024-09-15T03:42:50.864286+0000","last_fullsized":"2024-09-15T03:42:50.864286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:16.370806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:16.370806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:44.843373+0000","scrub_duration":0.0019016580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":153,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153580+0000","last_change":"2024-09-15T03:42:23.138835+0000","last_active":"2024-09-15T03:42:51.153580+0000","last_peered":"2024-09-15T03:42:51.153580+0000","last_clean":"2024-09-15T03:42:51.153580+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:42:51.153580+0000","last_undegraded":"2024-09-15T03:42:51.153580+0000","last_fullsized":"2024-09-15T03:42:51.153580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:04.333008+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657590+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:50.657590+0000","last_peered":"2024-09-15T03:42:50.657590+0000","last_clean":"2024-09-15T03:42:50.657590+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:50.657590+0000","last_undegraded":"2024-09-15T03:42:50.657590+0000","last_fullsized":"2024-09-15T03:42:50.657590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153579+0000","last_change":"2024-09-15T03:42:20.126436+0000","last_active":"2024-09-15T03:42:51.153579+0000","last_peered":"2024-09-15T03:42:51.153579+0000","last_clean":"2024-09-15T03:42:51.153579+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:42:51.153579+0000","last_undegraded":"2024-09-15T03:42:51.153579+0000","last_fullsized":"2024-09-15T03:42:51.153579+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:17.090770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":118,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.660207+0000","last_change":"2024-09-15T03:42:33.209224+0000","last_active":"2024-09-15T03:42:50.660207+0000","last_peered":"2024-09-15T03:42:50.660207+0000","last_clean":"2024-09-15T03:42:50.660207+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:42:50.660207+0000","last_undegraded":"2024-09-15T03:42:50.660207+0000","last_fullsized":"2024-09-15T03:42:50.660207+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T03:42:17.024578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:17.024578+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.982049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":269,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657603+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:50.657603+0000","last_peered":"2024-09-15T03:42:50.657603+0000","last_clean":"2024-09-15T03:42:50.657603+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:50.657603+0000","last_undegraded":"2024-09-15T03:42:50.657603+0000","last_fullsized":"2024-09-15T03:42:50.657603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304160,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067680,"statfs":{"total":96636764160,"available":96325304320,"internally_reserved":0,"allocated":2703360,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":1.202},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.96199999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":1.224},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.94699999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":1},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":1.0309999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.93899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.97899999999999998}]}]},{"osd":14,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.58199999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.75}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.626}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.83599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69999999999999996}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.81000000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.84599999999999997}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304164,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067676,"statfs":{"total":96636764160,"available":96325300224,"internally_reserved":0,"allocated":2711552,"data_stored":1816766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.504},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.624}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.88200000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.47199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.63400000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91300000000000003},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.84599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.73299999999999998},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.71099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.90000000000000002}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.94699999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.622},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.72599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.57599999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.0069999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.878}]}]},{"osd":6,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.88}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.625},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.81000000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.436},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.45900000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":54.079000000000001,"5min":54.079000000000001,"15min":54.079000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":909.03399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.52400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39400000000000002},{"interface":"front","average":{"1min":41.031999999999996,"5min":41.031999999999996,"15min":41.031999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":686.33799999999997,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.503}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73799999999999999}]}]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.72899999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48699999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.54800000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.61599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65600000000000003}]}]},{"osd":3,"up_from":68,"seq":292057776147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299556,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072284,"statfs":{"total":96636764160,"available":96330018816,"internally_reserved":0,"allocated":2187264,"data_stored":1288422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.73599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.625},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72099999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.622}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.501},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.442}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.53400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.42899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.40300000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.68999999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.60999999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.48399999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.80300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.501}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.68500000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":47.381,"5min":47.381,"15min":47.381},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":981.20500000000004,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53700000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89000000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.87},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755}]},{"osd":15,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.64100000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299468,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072372,"statfs":{"total":96636764160,"available":96330108928,"internally_reserved":0,"allocated":2154496,"data_stored":1269473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.84599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.61899999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.88600000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.97899999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.87}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.81899999999999995}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303624,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068216,"statfs":{"total":96636764160,"available":96325853184,"internally_reserved":0,"allocated":2150400,"data_stored":1269454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.78800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.54400000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.73499999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.51200000000000001},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.47999999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.58799999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.745}]}]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303632,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068208,"statfs":{"total":96636764160,"available":96325844992,"internally_reserved":0,"allocated":2154496,"data_stored":1270032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.42399999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.442}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.63100000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.502}]},{"osd":14,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":22.876000000000001,"5min":22.876000000000001,"15min":22.876000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":353.327,"5min":353.327,"15min":353.327},"last":0.57899999999999996},{"interface":"front","average":{"1min":61.741999999999997,"5min":61.741999999999997,"15min":61.741999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":974.78599999999994,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.61599999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66900000000000004}]}]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303648,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068192,"statfs":{"total":96636764160,"available":96325828608,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63700000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.752}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.41199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.78600000000000003}]},{"osd":13,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.623},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53800000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.57599999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.80700000000000005}]}]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82499999999999996},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":1.262}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":1.276},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96699999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.92800000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":1.2909999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":1.248}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.94899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.716},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.671}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.83999999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":1.232},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.98199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.70799999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.72999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":1},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":1.0640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":1.032},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":1.159}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.92800000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":1.1100000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":10.291,"5min":10.291,"15min":10.291},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":176.59299999999999,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":1.024},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.86099999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.748},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.82699999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.92300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69099999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91000000000000003},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.81100000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.079}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:56.289 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-15T03:42:56.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-15T03:42:56.591 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T03:42:56.601 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T03:42:56.601 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:42:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:56.881 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":144,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:42:55.349498+0000","last_up_change":"2024-09-15T03:41:35.013591+0000","last_in_change":"2024-09-15T03:42:20.111918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":46,"up_from":68,"up_thru":113,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43941},{"type":"v1","addr":"172.21.15.105:6803","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43941},{"type":"v1","addr":"172.21.15.105:6805","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43941},{"type":"v1","addr":"172.21.15.105:6809","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43941},{"type":"v1","addr":"172.21.15.105:6807","nonce":43941}]},"public_addr":"172.21.15.105:6803/43941","cluster_addr":"172.21.15.105:6805/43941","heartbeat_back_addr":"172.21.15.105:6809/43941","heartbeat_front_addr":"172.21.15.105:6807/43941","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:49.977726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.988005+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:46.256258+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.516008+0000","dead_epoch":47},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:09.575529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:43.785792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:04.563391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.889587+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:52.239895+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:02.352976+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:51.140062+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:41.474463+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:55.629737+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:54.451094+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.006408+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:51.897727+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:42:56.894 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:42:56.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:57.188 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T03:42:56.040357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4851136,"kb_used_data":35080,"kb_used_omap":26,"kb_used_meta":4815781,"kb_avail":1505098304,"statfs":{"total":1546188226560,"available":1541220663296,"internally_reserved":0,"allocated":35921920,"data_stored":21611197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4931360464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.726700"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.332176+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:51.332176+0000","last_peered":"2024-09-15T03:42:51.332176+0000","last_clean":"2024-09-15T03:42:51.332176+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:51.332176+0000","last_undegraded":"2024-09-15T03:42:51.332176+0000","last_fullsized":"2024-09-15T03:42:51.332176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.329944+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:51.329944+0000","last_peered":"2024-09-15T03:42:51.329944+0000","last_clean":"2024-09-15T03:42:51.329944+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:51.329944+0000","last_undegraded":"2024-09-15T03:42:51.329944+0000","last_fullsized":"2024-09-15T03:42:51.329944+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657364+0000","last_change":"2024-09-15T03:42:41.277510+0000","last_active":"2024-09-15T03:42:50.657364+0000","last_peered":"2024-09-15T03:42:50.657364+0000","last_clean":"2024-09-15T03:42:50.657364+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:42:50.657364+0000","last_undegraded":"2024-09-15T03:42:50.657364+0000","last_fullsized":"2024-09-15T03:42:50.657364+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.091680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.864286+0000","last_change":"2024-09-15T03:42:21.252540+0000","last_active":"2024-09-15T03:42:50.864286+0000","last_peered":"2024-09-15T03:42:50.864286+0000","last_clean":"2024-09-15T03:42:50.864286+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:42:50.864286+0000","last_undegraded":"2024-09-15T03:42:50.864286+0000","last_fullsized":"2024-09-15T03:42:50.864286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:16.370806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:16.370806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:44.843373+0000","scrub_duration":0.0019016580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":153,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153580+0000","last_change":"2024-09-15T03:42:23.138835+0000","last_active":"2024-09-15T03:42:51.153580+0000","last_peered":"2024-09-15T03:42:51.153580+0000","last_clean":"2024-09-15T03:42:51.153580+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:42:51.153580+0000","last_undegraded":"2024-09-15T03:42:51.153580+0000","last_fullsized":"2024-09-15T03:42:51.153580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:04.333008+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657590+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:50.657590+0000","last_peered":"2024-09-15T03:42:50.657590+0000","last_clean":"2024-09-15T03:42:50.657590+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:50.657590+0000","last_undegraded":"2024-09-15T03:42:50.657590+0000","last_fullsized":"2024-09-15T03:42:50.657590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153579+0000","last_change":"2024-09-15T03:42:20.126436+0000","last_active":"2024-09-15T03:42:51.153579+0000","last_peered":"2024-09-15T03:42:51.153579+0000","last_clean":"2024-09-15T03:42:51.153579+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:42:51.153579+0000","last_undegraded":"2024-09-15T03:42:51.153579+0000","last_fullsized":"2024-09-15T03:42:51.153579+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:17.090770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":118,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.660207+0000","last_change":"2024-09-15T03:42:33.209224+0000","last_active":"2024-09-15T03:42:50.660207+0000","last_peered":"2024-09-15T03:42:50.660207+0000","last_clean":"2024-09-15T03:42:50.660207+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:42:50.660207+0000","last_undegraded":"2024-09-15T03:42:50.660207+0000","last_fullsized":"2024-09-15T03:42:50.660207+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T03:42:17.024578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:17.024578+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.982049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":269,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657603+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:50.657603+0000","last_peered":"2024-09-15T03:42:50.657603+0000","last_clean":"2024-09-15T03:42:50.657603+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:50.657603+0000","last_undegraded":"2024-09-15T03:42:50.657603+0000","last_fullsized":"2024-09-15T03:42:50.657603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304160,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067680,"statfs":{"total":96636764160,"available":96325304320,"internally_reserved":0,"allocated":2703360,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":1.202},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.96199999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":1.224},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.94699999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":1},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":1.0309999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.93899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.97899999999999998}]}]},{"osd":14,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.58199999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.75}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.626}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.83599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69999999999999996}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.81000000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.84599999999999997}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304164,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067676,"statfs":{"total":96636764160,"available":96325300224,"internally_reserved":0,"allocated":2711552,"data_stored":1816766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.504},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.624}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.88200000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.47199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.63400000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91300000000000003},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.84599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.73299999999999998},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.71099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.90000000000000002}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.94699999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.622},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.72599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.57599999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.0069999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.878}]}]},{"osd":6,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.88}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.625},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.81000000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.436},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.45900000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":54.079000000000001,"5min":54.079000000000001,"15min":54.079000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":909.03399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.52400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39400000000000002},{"interface":"front","average":{"1min":41.031999999999996,"5min":41.031999999999996,"15min":41.031999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":686.33799999999997,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.503}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73799999999999999}]}]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.72899999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48699999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.54800000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.61599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65600000000000003}]}]},{"osd":3,"up_from":68,"seq":292057776147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299556,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072284,"statfs":{"total":96636764160,"available":96330018816,"internally_reserved":0,"allocated":2187264,"data_stored":1288422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.73599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.625},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72099999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.622}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.501},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.442}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.53400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.42899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.40300000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.68999999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.60999999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.48399999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.80300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.501}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.68500000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":47.381,"5min":47.381,"15min":47.381},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":981.20500000000004,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53700000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89000000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.87},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755}]},{"osd":15,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.64100000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299468,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072372,"statfs":{"total":96636764160,"available":96330108928,"internally_reserved":0,"allocated":2154496,"data_stored":1269473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.84599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.61899999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.88600000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.97899999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.87}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.81899999999999995}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303624,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068216,"statfs":{"total":96636764160,"available":96325853184,"internally_reserved":0,"allocated":2150400,"data_stored":1269454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.78800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.54400000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.73499999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.51200000000000001},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.47999999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.58799999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.745}]}]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303632,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068208,"statfs":{"total":96636764160,"available":96325844992,"internally_reserved":0,"allocated":2154496,"data_stored":1270032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.42399999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.442}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.63100000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.502}]},{"osd":14,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":22.876000000000001,"5min":22.876000000000001,"15min":22.876000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":353.327,"5min":353.327,"15min":353.327},"last":0.57899999999999996},{"interface":"front","average":{"1min":61.741999999999997,"5min":61.741999999999997,"15min":61.741999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":974.78599999999994,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.61599999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66900000000000004}]}]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303648,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068192,"statfs":{"total":96636764160,"available":96325828608,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63700000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.752}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.41199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.78600000000000003}]},{"osd":13,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.623},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53800000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.57599999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.80700000000000005}]}]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82499999999999996},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":1.262}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":1.276},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96699999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.92800000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":1.2909999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":1.248}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.94899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.716},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.671}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.83999999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":1.232},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.98199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.70799999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.72999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":1},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":1.0640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":1.032},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":1.159}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.92800000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":1.1100000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":10.291,"5min":10.291,"15min":10.291},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":176.59299999999999,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":1.024},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.86099999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.748},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.82699999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.92300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69099999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91000000000000003},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.81100000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.079}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:57.193 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:57.477 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:57.477 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:57.488 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T03:42:56.040357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4851136,"kb_used_data":35080,"kb_used_omap":26,"kb_used_meta":4815781,"kb_avail":1505098304,"statfs":{"total":1546188226560,"available":1541220663296,"internally_reserved":0,"allocated":35921920,"data_stored":21611197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4931360464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.726700"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.332176+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:51.332176+0000","last_peered":"2024-09-15T03:42:51.332176+0000","last_clean":"2024-09-15T03:42:51.332176+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:51.332176+0000","last_undegraded":"2024-09-15T03:42:51.332176+0000","last_fullsized":"2024-09-15T03:42:51.332176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.329944+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:51.329944+0000","last_peered":"2024-09-15T03:42:51.329944+0000","last_clean":"2024-09-15T03:42:51.329944+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:51.329944+0000","last_undegraded":"2024-09-15T03:42:51.329944+0000","last_fullsized":"2024-09-15T03:42:51.329944+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657364+0000","last_change":"2024-09-15T03:42:41.277510+0000","last_active":"2024-09-15T03:42:50.657364+0000","last_peered":"2024-09-15T03:42:50.657364+0000","last_clean":"2024-09-15T03:42:50.657364+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:42:50.657364+0000","last_undegraded":"2024-09-15T03:42:50.657364+0000","last_fullsized":"2024-09-15T03:42:50.657364+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.091680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.864286+0000","last_change":"2024-09-15T03:42:21.252540+0000","last_active":"2024-09-15T03:42:50.864286+0000","last_peered":"2024-09-15T03:42:50.864286+0000","last_clean":"2024-09-15T03:42:50.864286+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:42:50.864286+0000","last_undegraded":"2024-09-15T03:42:50.864286+0000","last_fullsized":"2024-09-15T03:42:50.864286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:16.370806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:16.370806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:44.843373+0000","scrub_duration":0.0019016580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":153,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153580+0000","last_change":"2024-09-15T03:42:23.138835+0000","last_active":"2024-09-15T03:42:51.153580+0000","last_peered":"2024-09-15T03:42:51.153580+0000","last_clean":"2024-09-15T03:42:51.153580+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:42:51.153580+0000","last_undegraded":"2024-09-15T03:42:51.153580+0000","last_fullsized":"2024-09-15T03:42:51.153580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:04.333008+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657590+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:50.657590+0000","last_peered":"2024-09-15T03:42:50.657590+0000","last_clean":"2024-09-15T03:42:50.657590+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:50.657590+0000","last_undegraded":"2024-09-15T03:42:50.657590+0000","last_fullsized":"2024-09-15T03:42:50.657590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153579+0000","last_change":"2024-09-15T03:42:20.126436+0000","last_active":"2024-09-15T03:42:51.153579+0000","last_peered":"2024-09-15T03:42:51.153579+0000","last_clean":"2024-09-15T03:42:51.153579+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:42:51.153579+0000","last_undegraded":"2024-09-15T03:42:51.153579+0000","last_fullsized":"2024-09-15T03:42:51.153579+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:17.090770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":118,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.660207+0000","last_change":"2024-09-15T03:42:33.209224+0000","last_active":"2024-09-15T03:42:50.660207+0000","last_peered":"2024-09-15T03:42:50.660207+0000","last_clean":"2024-09-15T03:42:50.660207+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:42:50.660207+0000","last_undegraded":"2024-09-15T03:42:50.660207+0000","last_fullsized":"2024-09-15T03:42:50.660207+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T03:42:17.024578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:17.024578+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.982049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":269,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657603+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:50.657603+0000","last_peered":"2024-09-15T03:42:50.657603+0000","last_clean":"2024-09-15T03:42:50.657603+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:50.657603+0000","last_undegraded":"2024-09-15T03:42:50.657603+0000","last_fullsized":"2024-09-15T03:42:50.657603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304160,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067680,"statfs":{"total":96636764160,"available":96325304320,"internally_reserved":0,"allocated":2703360,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":1.202},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.96199999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":1.224},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.94699999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":1},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":1.0309999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.93899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.97899999999999998}]}]},{"osd":14,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.58199999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.75}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.626}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.83599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69999999999999996}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.81000000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.84599999999999997}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304164,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067676,"statfs":{"total":96636764160,"available":96325300224,"internally_reserved":0,"allocated":2711552,"data_stored":1816766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.504},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.624}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.88200000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.47199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.63400000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91300000000000003},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.84599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.73299999999999998},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.71099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.90000000000000002}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.94699999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.622},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.72599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.57599999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.0069999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.878}]}]},{"osd":6,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.88}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.625},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.81000000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.436},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.45900000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":54.079000000000001,"5min":54.079000000000001,"15min":54.079000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":909.03399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.52400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39400000000000002},{"interface":"front","average":{"1min":41.031999999999996,"5min":41.031999999999996,"15min":41.031999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":686.33799999999997,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.503}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73799999999999999}]}]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.72899999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48699999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.54800000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.61599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65600000000000003}]}]},{"osd":3,"up_from":68,"seq":292057776147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299556,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072284,"statfs":{"total":96636764160,"available":96330018816,"internally_reserved":0,"allocated":2187264,"data_stored":1288422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.73599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.625},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72099999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.622}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.501},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.442}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.53400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.42899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.40300000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.68999999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.60999999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.48399999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.80300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.501}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.68500000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":47.381,"5min":47.381,"15min":47.381},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":981.20500000000004,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53700000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89000000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.87},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755}]},{"osd":15,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.64100000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299468,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072372,"statfs":{"total":96636764160,"available":96330108928,"internally_reserved":0,"allocated":2154496,"data_stored":1269473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.84599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.61899999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.88600000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.97899999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.87}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.81899999999999995}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303624,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068216,"statfs":{"total":96636764160,"available":96325853184,"internally_reserved":0,"allocated":2150400,"data_stored":1269454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.78800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.54400000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.73499999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.51200000000000001},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.47999999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.58799999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.745}]}]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303632,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068208,"statfs":{"total":96636764160,"available":96325844992,"internally_reserved":0,"allocated":2154496,"data_stored":1270032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.42399999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.442}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.63100000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.502}]},{"osd":14,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":22.876000000000001,"5min":22.876000000000001,"15min":22.876000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":353.327,"5min":353.327,"15min":353.327},"last":0.57899999999999996},{"interface":"front","average":{"1min":61.741999999999997,"5min":61.741999999999997,"15min":61.741999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":974.78599999999994,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.61599999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66900000000000004}]}]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303648,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068192,"statfs":{"total":96636764160,"available":96325828608,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63700000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.752}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.41199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.78600000000000003}]},{"osd":13,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.623},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53800000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.57599999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.80700000000000005}]}]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82499999999999996},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":1.262}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":1.276},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96699999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.92800000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":1.2909999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":1.248}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.94899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.716},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.671}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.83999999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":1.232},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.98199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.70799999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.72999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":1},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":1.0640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":1.032},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":1.159}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.92800000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":1.1100000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":10.291,"5min":10.291,"15min":10.291},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":176.59299999999999,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":1.024},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.86099999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.748},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.82699999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.92300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69099999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91000000000000003},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.81100000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.079}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:57.493 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:42:57.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:42:57.773 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:42:57.785 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T03:42:56.040357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4851136,"kb_used_data":35080,"kb_used_omap":26,"kb_used_meta":4815781,"kb_avail":1505098304,"statfs":{"total":1546188226560,"available":1541220663296,"internally_reserved":0,"allocated":35921920,"data_stored":21611197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4931360464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.726700"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.332176+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:51.332176+0000","last_peered":"2024-09-15T03:42:51.332176+0000","last_clean":"2024-09-15T03:42:51.332176+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:51.332176+0000","last_undegraded":"2024-09-15T03:42:51.332176+0000","last_fullsized":"2024-09-15T03:42:51.332176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":138,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.329944+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:42:51.329944+0000","last_peered":"2024-09-15T03:42:51.329944+0000","last_clean":"2024-09-15T03:42:51.329944+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:42:51.329944+0000","last_undegraded":"2024-09-15T03:42:51.329944+0000","last_fullsized":"2024-09-15T03:42:51.329944+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.509622+0000","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657364+0000","last_change":"2024-09-15T03:42:41.277510+0000","last_active":"2024-09-15T03:42:50.657364+0000","last_peered":"2024-09-15T03:42:50.657364+0000","last_clean":"2024-09-15T03:42:50.657364+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:42:50.657364+0000","last_undegraded":"2024-09-15T03:42:50.657364+0000","last_fullsized":"2024-09-15T03:42:50.657364+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.091680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.864286+0000","last_change":"2024-09-15T03:42:21.252540+0000","last_active":"2024-09-15T03:42:50.864286+0000","last_peered":"2024-09-15T03:42:50.864286+0000","last_clean":"2024-09-15T03:42:50.864286+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:42:50.864286+0000","last_undegraded":"2024-09-15T03:42:50.864286+0000","last_fullsized":"2024-09-15T03:42:50.864286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:16.370806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:16.370806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:44.843373+0000","scrub_duration":0.0019016580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":153,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153580+0000","last_change":"2024-09-15T03:42:23.138835+0000","last_active":"2024-09-15T03:42:51.153580+0000","last_peered":"2024-09-15T03:42:51.153580+0000","last_clean":"2024-09-15T03:42:51.153580+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:42:51.153580+0000","last_undegraded":"2024-09-15T03:42:51.153580+0000","last_fullsized":"2024-09-15T03:42:51.153580+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:04.333008+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657590+0000","last_change":"2024-09-15T03:41:56.830612+0000","last_active":"2024-09-15T03:42:50.657590+0000","last_peered":"2024-09-15T03:42:50.657590+0000","last_clean":"2024-09-15T03:42:50.657590+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:42:50.657590+0000","last_undegraded":"2024-09-15T03:42:50.657590+0000","last_fullsized":"2024-09-15T03:42:50.657590+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:41:56.830512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:56.830512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.602533+0000","scrub_duration":0.001534728,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:51.153579+0000","last_change":"2024-09-15T03:42:20.126436+0000","last_active":"2024-09-15T03:42:51.153579+0000","last_peered":"2024-09-15T03:42:51.153579+0000","last_clean":"2024-09-15T03:42:51.153579+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:42:51.153579+0000","last_undegraded":"2024-09-15T03:42:51.153579+0000","last_fullsized":"2024-09-15T03:42:51.153579+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:17.090770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":118,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.660207+0000","last_change":"2024-09-15T03:42:33.209224+0000","last_active":"2024-09-15T03:42:50.660207+0000","last_peered":"2024-09-15T03:42:50.660207+0000","last_clean":"2024-09-15T03:42:50.660207+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:42:50.660207+0000","last_undegraded":"2024-09-15T03:42:50.660207+0000","last_fullsized":"2024-09-15T03:42:50.660207+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'1","last_scrub_stamp":"2024-09-15T03:42:17.024578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:17.024578+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.982049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":269,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-15T03:42:50.657603+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:50.657603+0000","last_peered":"2024-09-15T03:42:50.657603+0000","last_clean":"2024-09-15T03:42:50.657603+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:50.657603+0000","last_undegraded":"2024-09-15T03:42:50.657603+0000","last_fullsized":"2024-09-15T03:42:50.657603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:00.688636+0000","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304160,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067680,"statfs":{"total":96636764160,"available":96325304320,"internally_reserved":0,"allocated":2703360,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":1.202},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.96199999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":1.224},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.94699999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":1},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":1.0309999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.93899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.97899999999999998}]}]},{"osd":14,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.58199999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.75}]},{"osd":8,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.626}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.60899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.83599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69999999999999996}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.81000000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.84599999999999997}]}]},{"osd":13,"up_from":22,"seq":94489280541,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304164,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067676,"statfs":{"total":96636764160,"available":96325300224,"internally_reserved":0,"allocated":2711552,"data_stored":1816766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.504},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.624}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.80900000000000005},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.88200000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.47199999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.63400000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.91300000000000003},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.84599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.73299999999999998},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.71099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.90000000000000002}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.94699999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.622},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.72599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.96999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.749}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.57599999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.59899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79700000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.0069999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.878}]}]},{"osd":6,"up_from":21,"seq":90194313245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.88}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.64800000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.625},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.75600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.81000000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.436},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.45900000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":54.079000000000001,"5min":54.079000000000001,"15min":54.079000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":909.03399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.52400000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39400000000000002},{"interface":"front","average":{"1min":41.031999999999996,"5min":41.031999999999996,"15min":41.031999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":686.33799999999997,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.503}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73799999999999999}]}]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303652,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.72899999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48699999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.54800000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.61599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.65600000000000003}]}]},{"osd":3,"up_from":68,"seq":292057776147,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299556,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072284,"statfs":{"total":96636764160,"available":96330018816,"internally_reserved":0,"allocated":2187264,"data_stored":1288422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.73599999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.625},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72099999999999997}]}]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.622}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.501},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.442}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.53400000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70199999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.42899999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.40300000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.68999999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.60999999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.48399999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.80300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.501}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.68500000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":47.381,"5min":47.381,"15min":47.381},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":981.20500000000004,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.53700000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89000000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.87},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.77200000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.755}]},{"osd":15,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.64100000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299468,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072372,"statfs":{"total":96636764160,"available":96330108928,"internally_reserved":0,"allocated":2154496,"data_stored":1269473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.84599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.61899999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.78700000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.88600000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.97899999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.87}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.81899999999999995}]}]},{"osd":1,"up_from":22,"seq":94489280541,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303624,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068216,"statfs":{"total":96636764160,"available":96325853184,"internally_reserved":0,"allocated":2150400,"data_stored":1269454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.78800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.54400000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.73499999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.51200000000000001},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.47999999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.58799999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.745}]}]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303632,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068208,"statfs":{"total":96636764160,"available":96325844992,"internally_reserved":0,"allocated":2154496,"data_stored":1270032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.42399999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.442}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.63100000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.502}]},{"osd":14,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":22.876000000000001,"5min":22.876000000000001,"15min":22.876000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":353.327,"5min":353.327,"15min":353.327},"last":0.57899999999999996},{"interface":"front","average":{"1min":61.741999999999997,"5min":61.741999999999997,"15min":61.741999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":974.78599999999994,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.61599999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66900000000000004}]}]},{"osd":5,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303648,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068192,"statfs":{"total":96636764160,"available":96325828608,"internally_reserved":0,"allocated":2183168,"data_stored":1288452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63700000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.752}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.41199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.78600000000000003}]},{"osd":13,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.623},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53800000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.57599999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.80700000000000005}]}]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82499999999999996},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":1.262}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":1.276},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96699999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78000000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.92800000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":1.2909999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":1.248}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.94899999999999995}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.716},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.671}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.83999999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":1.232},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.98199999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068184,"statfs":{"total":96636764160,"available":96325820416,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.83899999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.70799999999999996}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.72999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":1},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":1.0640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":1.032},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.79900000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":1.159}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.92800000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":1.1100000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303652,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068188,"statfs":{"total":96636764160,"available":96325824512,"internally_reserved":0,"allocated":2187264,"data_stored":1289030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":10.291,"5min":10.291,"15min":10.291},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":176.59299999999999,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":1.024},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.86099999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.748},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45800000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.82699999999999996}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.92300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69099999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91000000000000003},{"interface":"front","average":{"1min":51.904000000000003,"5min":51.904000000000003,"15min":51.904000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":923.202,"5min":923.202,"15min":923.202},"last":0.81100000000000005}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.079}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:42:57.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:42:58.067 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:42:58.068 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":144,"fsid":"da8e668f-207e-4717-9e66-7070b1986601","created":"2024-09-15T03:40:30.211709+0000","modified":"2024-09-15T03:42:55.349498+0000","last_up_change":"2024-09-15T03:41:35.013591+0000","last_in_change":"2024-09-15T03:42:20.111918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:40:42.070557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:40:44.856082+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":29,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2efef372-d786-40e5-96a1-10f399314942","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33102},{"type":"v1","addr":"172.21.15.5:6801","nonce":33102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33102},{"type":"v1","addr":"172.21.15.5:6803","nonce":33102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33102},{"type":"v1","addr":"172.21.15.5:6807","nonce":33102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33102},{"type":"v1","addr":"172.21.15.5:6805","nonce":33102}]},"public_addr":"172.21.15.5:6801/33102","cluster_addr":"172.21.15.5:6803/33102","heartbeat_back_addr":"172.21.15.5:6807/33102","heartbeat_front_addr":"172.21.15.5:6805/33102","state":["exists","up"]},{"osd":1,"uuid":"d77b5a7a-5228-4bed-b89b-799b0d0b9491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":33259},{"type":"v1","addr":"172.21.15.64:6812","nonce":33259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":33259},{"type":"v1","addr":"172.21.15.64:6816","nonce":33259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":33259},{"type":"v1","addr":"172.21.15.64:6824","nonce":33259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":33259},{"type":"v1","addr":"172.21.15.64:6820","nonce":33259}]},"public_addr":"172.21.15.64:6812/33259","cluster_addr":"172.21.15.64:6816/33259","heartbeat_back_addr":"172.21.15.64:6824/33259","heartbeat_front_addr":"172.21.15.64:6820/33259","state":["exists","up"]},{"osd":2,"uuid":"e81da5de-c743-4cf0-a36a-beee634f01da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":32810},{"type":"v1","addr":"172.21.15.78:6811","nonce":32810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":32810},{"type":"v1","addr":"172.21.15.78:6815","nonce":32810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":32810},{"type":"v1","addr":"172.21.15.78:6823","nonce":32810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":32810},{"type":"v1","addr":"172.21.15.78:6819","nonce":32810}]},"public_addr":"172.21.15.78:6811/32810","cluster_addr":"172.21.15.78:6815/32810","heartbeat_back_addr":"172.21.15.78:6823/32810","heartbeat_front_addr":"172.21.15.78:6819/32810","state":["exists","up"]},{"osd":3,"uuid":"dae3f484-9f24-4f24-96c3-253465d0d369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":46,"up_from":68,"up_thru":113,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43941},{"type":"v1","addr":"172.21.15.105:6803","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43941},{"type":"v1","addr":"172.21.15.105:6805","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43941},{"type":"v1","addr":"172.21.15.105:6809","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43941},{"type":"v1","addr":"172.21.15.105:6807","nonce":43941}]},"public_addr":"172.21.15.105:6803/43941","cluster_addr":"172.21.15.105:6805/43941","heartbeat_back_addr":"172.21.15.105:6809/43941","heartbeat_front_addr":"172.21.15.105:6807/43941","state":["exists","up"]},{"osd":4,"uuid":"7a2d4b3d-e398-4010-8952-4e5207e005b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33104},{"type":"v1","addr":"172.21.15.5:6817","nonce":33104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33104},{"type":"v1","addr":"172.21.15.5:6819","nonce":33104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33104},{"type":"v1","addr":"172.21.15.5:6823","nonce":33104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33104},{"type":"v1","addr":"172.21.15.5:6821","nonce":33104}]},"public_addr":"172.21.15.5:6817/33104","cluster_addr":"172.21.15.5:6819/33104","heartbeat_back_addr":"172.21.15.5:6823/33104","heartbeat_front_addr":"172.21.15.5:6821/33104","state":["exists","up"]},{"osd":5,"uuid":"b757a5a4-3278-42ec-b43f-6c08bfc15f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":33258},{"type":"v1","addr":"172.21.15.64:6801","nonce":33258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":33258},{"type":"v1","addr":"172.21.15.64:6803","nonce":33258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":33258},{"type":"v1","addr":"172.21.15.64:6807","nonce":33258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":33258},{"type":"v1","addr":"172.21.15.64:6805","nonce":33258}]},"public_addr":"172.21.15.64:6801/33258","cluster_addr":"172.21.15.64:6803/33258","heartbeat_back_addr":"172.21.15.64:6807/33258","heartbeat_front_addr":"172.21.15.64:6805/33258","state":["exists","up"]},{"osd":6,"uuid":"96572944-7f1f-4461-a818-89bd0c4e2faf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":32809},{"type":"v1","addr":"172.21.15.78:6801","nonce":32809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":32809},{"type":"v1","addr":"172.21.15.78:6803","nonce":32809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":32809},{"type":"v1","addr":"172.21.15.78:6807","nonce":32809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":32809},{"type":"v1","addr":"172.21.15.78:6805","nonce":32809}]},"public_addr":"172.21.15.78:6801/32809","cluster_addr":"172.21.15.78:6803/32809","heartbeat_back_addr":"172.21.15.78:6807/32809","heartbeat_front_addr":"172.21.15.78:6805/32809","state":["exists","up"]},{"osd":7,"uuid":"e698cf1c-5b5d-43a2-b44a-3d30af6f5917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":33114},{"type":"v1","addr":"172.21.15.105:6814","nonce":33114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":33114},{"type":"v1","addr":"172.21.15.105:6819","nonce":33114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":33114},{"type":"v1","addr":"172.21.15.105:6825","nonce":33114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6821","nonce":33114},{"type":"v1","addr":"172.21.15.105:6823","nonce":33114}]},"public_addr":"172.21.15.105:6814/33114","cluster_addr":"172.21.15.105:6819/33114","heartbeat_back_addr":"172.21.15.105:6825/33114","heartbeat_front_addr":"172.21.15.105:6823/33114","state":["exists","up"]},{"osd":8,"uuid":"12a65fa6-1fbd-4900-88c9-69695141bc1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33107},{"type":"v1","addr":"172.21.15.5:6809","nonce":33107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33107},{"type":"v1","addr":"172.21.15.5:6811","nonce":33107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33107},{"type":"v1","addr":"172.21.15.5:6815","nonce":33107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33107},{"type":"v1","addr":"172.21.15.5:6813","nonce":33107}]},"public_addr":"172.21.15.5:6809/33107","cluster_addr":"172.21.15.5:6811/33107","heartbeat_back_addr":"172.21.15.5:6815/33107","heartbeat_front_addr":"172.21.15.5:6813/33107","state":["exists","up"]},{"osd":9,"uuid":"8bbc59be-d8eb-49a8-8c9f-07dbc78f4fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":33260},{"type":"v1","addr":"172.21.15.64:6809","nonce":33260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":33260},{"type":"v1","addr":"172.21.15.64:6813","nonce":33260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6819","nonce":33260},{"type":"v1","addr":"172.21.15.64:6821","nonce":33260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":33260},{"type":"v1","addr":"172.21.15.64:6817","nonce":33260}]},"public_addr":"172.21.15.64:6809/33260","cluster_addr":"172.21.15.64:6813/33260","heartbeat_back_addr":"172.21.15.64:6821/33260","heartbeat_front_addr":"172.21.15.64:6817/33260","state":["exists","up"]},{"osd":10,"uuid":"91199e3c-694d-4888-859b-b76feac7c96d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":32811},{"type":"v1","addr":"172.21.15.78:6825","nonce":32811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":32811},{"type":"v1","addr":"172.21.15.78:6827","nonce":32811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":32811},{"type":"v1","addr":"172.21.15.78:6831","nonce":32811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":32811},{"type":"v1","addr":"172.21.15.78:6829","nonce":32811}]},"public_addr":"172.21.15.78:6825/32811","cluster_addr":"172.21.15.78:6827/32811","heartbeat_back_addr":"172.21.15.78:6831/32811","heartbeat_front_addr":"172.21.15.78:6829/32811","state":["exists","up"]},{"osd":11,"uuid":"4d6ca3aa-f880-4673-8d66-21d438dd4b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":33115},{"type":"v1","addr":"172.21.15.105:6811","nonce":33115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":33115},{"type":"v1","addr":"172.21.15.105:6815","nonce":33115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":33115},{"type":"v1","addr":"172.21.15.105:6822","nonce":33115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6817","nonce":33115},{"type":"v1","addr":"172.21.15.105:6818","nonce":33115}]},"public_addr":"172.21.15.105:6811/33115","cluster_addr":"172.21.15.105:6815/33115","heartbeat_back_addr":"172.21.15.105:6822/33115","heartbeat_front_addr":"172.21.15.105:6818/33115","state":["exists","up"]},{"osd":12,"uuid":"5d4556fd-7659-4e2b-b60b-e44eca29c3e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33108},{"type":"v1","addr":"172.21.15.5:6825","nonce":33108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":33108},{"type":"v1","addr":"172.21.15.5:6827","nonce":33108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":33108},{"type":"v1","addr":"172.21.15.5:6831","nonce":33108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":33108},{"type":"v1","addr":"172.21.15.5:6829","nonce":33108}]},"public_addr":"172.21.15.5:6825/33108","cluster_addr":"172.21.15.5:6827/33108","heartbeat_back_addr":"172.21.15.5:6831/33108","heartbeat_front_addr":"172.21.15.5:6829/33108","state":["exists","up"]},{"osd":13,"uuid":"78791966-b232-400f-9c30-cabebfa73dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6823","nonce":33261},{"type":"v1","addr":"172.21.15.64:6825","nonce":33261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":33261},{"type":"v1","addr":"172.21.15.64:6827","nonce":33261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":33261},{"type":"v1","addr":"172.21.15.64:6831","nonce":33261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":33261},{"type":"v1","addr":"172.21.15.64:6829","nonce":33261}]},"public_addr":"172.21.15.64:6825/33261","cluster_addr":"172.21.15.64:6827/33261","heartbeat_back_addr":"172.21.15.64:6831/33261","heartbeat_front_addr":"172.21.15.64:6829/33261","state":["exists","up"]},{"osd":14,"uuid":"4a275057-7e86-452f-958d-80263088db83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":32812},{"type":"v1","addr":"172.21.15.78:6810","nonce":32812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":32812},{"type":"v1","addr":"172.21.15.78:6814","nonce":32812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":32812},{"type":"v1","addr":"172.21.15.78:6822","nonce":32812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":32812},{"type":"v1","addr":"172.21.15.78:6818","nonce":32812}]},"public_addr":"172.21.15.78:6810/32812","cluster_addr":"172.21.15.78:6814/32812","heartbeat_back_addr":"172.21.15.78:6822/32812","heartbeat_front_addr":"172.21.15.78:6818/32812","state":["exists","up"]},{"osd":15,"uuid":"18589d26-e121-4e72-bf6e-24a4a37c49c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":33116},{"type":"v1","addr":"172.21.15.105:6827","nonce":33116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":33116},{"type":"v1","addr":"172.21.15.105:6829","nonce":33116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":33116},{"type":"v1","addr":"172.21.15.105:6833","nonce":33116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":33116},{"type":"v1","addr":"172.21.15.105:6831","nonce":33116}]},"public_addr":"172.21.15.105:6827/33116","cluster_addr":"172.21.15.105:6829/33116","heartbeat_back_addr":"172.21.15.105:6833/33116","heartbeat_front_addr":"172.21.15.105:6831/33116","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:49.977726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.988005+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:46.256258+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:08.516008+0000","dead_epoch":47},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:09.575529+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:43.785792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:04.563391+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.889587+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:52.239895+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:02.352976+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:51.140062+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:41.474463+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:55.629737+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:54.451094+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:42:01.006408+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:41:51.897727+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:42:59.080 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T03:42:59.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:42:59.229 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:42:59.229 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:42:59.229 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:42:59.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T03:42:59.520 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T03:42:59.532 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T03:42:59.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:42:59.664 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:42:59.664 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:42:59.664 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:42:59.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T03:42:59.951 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T03:42:59.964 INFO:tasks.ceph:Scrubbing osd.8 2024-09-15T03:42:59.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:00.095 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:00.095 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:00.106 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-15T03:43:00.389 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 8 to deep-scrub 2024-09-15T03:43:00.402 INFO:tasks.ceph:Scrubbing osd.12 2024-09-15T03:43:00.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:00.532 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:00.533 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:00.533 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:00.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-15T03:43:00.818 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 12 to deep-scrub 2024-09-15T03:43:00.831 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T03:43:00.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:00.963 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:00.963 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:00.963 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:00.973 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T03:43:01.249 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T03:43:01.262 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T03:43:01.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:01.394 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:01.394 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:01.395 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:01.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T03:43:01.684 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T03:43:01.697 INFO:tasks.ceph:Scrubbing osd.9 2024-09-15T03:43:01.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:01.828 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:01.828 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:01.828 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:01.839 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-15T03:43:02.118 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 9 to deep-scrub 2024-09-15T03:43:02.131 INFO:tasks.ceph:Scrubbing osd.13 2024-09-15T03:43:02.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:02.263 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:02.263 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:02.263 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:02.273 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-15T03:43:02.554 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 13 to deep-scrub 2024-09-15T03:43:02.566 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T03:43:02.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:02.698 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:02.698 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:02.699 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:02.709 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T03:43:02.986 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T03:43:02.999 INFO:tasks.ceph:Scrubbing osd.6 2024-09-15T03:43:02.999 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:03.129 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:03.129 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:03.129 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:03.139 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-15T03:43:03.418 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 6 to deep-scrub 2024-09-15T03:43:03.431 INFO:tasks.ceph:Scrubbing osd.10 2024-09-15T03:43:03.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:03.563 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:03.563 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:03.563 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:03.573 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-15T03:43:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 10 to deep-scrub 2024-09-15T03:43:04.704 INFO:tasks.ceph:Scrubbing osd.14 2024-09-15T03:43:04.704 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:04.842 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:04.842 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:04.843 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:04.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-15T03:43:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 14 to deep-scrub 2024-09-15T03:43:05.143 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T03:43:05.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:05.274 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:05.275 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:05.275 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:05.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T03:43:05.568 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T03:43:05.582 INFO:tasks.ceph:Scrubbing osd.7 2024-09-15T03:43:05.582 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:05.713 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:05.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-15T03:43:06.000 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 7 to deep-scrub 2024-09-15T03:43:06.011 INFO:tasks.ceph:Scrubbing osd.11 2024-09-15T03:43:06.011 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:06.142 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:06.143 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:06.143 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:06.153 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-15T03:43:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 11 to deep-scrub 2024-09-15T03:43:06.445 INFO:tasks.ceph:Scrubbing osd.15 2024-09-15T03:43:06.445 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:43:06.576 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-15T03:43:06.576 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:43:06.576 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-15T03:43:06.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-15T03:43:06.866 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 15 to deep-scrub 2024-09-15T03:43:06.881 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:43:07.163 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:43:07.164 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:43:07.175 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T03:43:06.042563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4851884,"kb_used_data":35880,"kb_used_omap":26,"kb_used_meta":4815781,"kb_avail":1505097556,"statfs":{"total":1546188226560,"available":1541219897344,"internally_reserved":0,"allocated":36741120,"data_stored":22086853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4931360464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001818"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":137,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:42:59.539048+0000","last_change":"2024-09-15T03:42:05.044190+0000","last_active":"2024-09-15T03:42:59.539048+0000","last_peered":"2024-09-15T03:42:59.539048+0000","last_clean":"2024-09-15T03:42:59.539048+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:42:59.539048+0000","last_undegraded":"2024-09-15T03:42:59.539048+0000","last_fullsized":"2024-09-15T03:42:59.539048+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.044126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.044126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:15.987307+0000","scrub_duration":0.0012804170000000001,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:43:00.389518+0000","last_change":"2024-09-15T03:42:05.088097+0000","last_active":"2024-09-15T03:43:00.389518+0000","last_peered":"2024-09-15T03:43:00.389518+0000","last_clean":"2024-09-15T03:43:00.389518+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:43:00.389518+0000","last_undegraded":"2024-09-15T03:43:00.389518+0000","last_fullsized":"2024-09-15T03:43:00.389518+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.088015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.088015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0016149409999999999,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:42:59.784004+0000","last_change":"2024-09-15T03:42:41.277510+0000","last_active":"2024-09-15T03:42:59.784004+0000","last_peered":"2024-09-15T03:42:59.784004+0000","last_clean":"2024-09-15T03:42:59.784004+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:42:59.784004+0000","last_undegraded":"2024-09-15T03:42:59.784004+0000","last_fullsized":"2024-09-15T03:42:59.784004+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:07.145656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:07.145656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:20.091680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:43:00.339462+0000","last_change":"2024-09-15T03:43:00.339427+0000","last_active":"2024-09-15T03:43:00.339462+0000","last_peered":"2024-09-15T03:43:00.339462+0000","last_clean":"2024-09-15T03:43:00.339462+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:43:00.339462+0000","last_undegraded":"2024-09-15T03:43:00.339462+0000","last_fullsized":"2024-09-15T03:43:00.339462+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:00.339350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:43:00.339350+0000","last_clean_scrub_stamp":"2024-09-15T03:43:00.339350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:22.051499+0000","scrub_duration":0.0012992660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'2","reported_seq":156,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:43:00.057271+0000","last_change":"2024-09-15T03:42:23.138835+0000","last_active":"2024-09-15T03:43:00.057271+0000","last_peered":"2024-09-15T03:43:00.057271+0000","last_clean":"2024-09-15T03:43:00.057271+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:43:00.057271+0000","last_undegraded":"2024-09-15T03:43:00.057271+0000","last_fullsized":"2024-09-15T03:43:00.057271+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-09-15T03:41:55.835634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:41:55.835634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:04.333008+0000","scrub_duration":0.001745202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:43:01.957951+0000","last_change":"2024-09-15T03:43:01.957922+0000","last_active":"2024-09-15T03:43:01.957951+0000","last_peered":"2024-09-15T03:43:01.957951+0000","last_clean":"2024-09-15T03:43:01.957951+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:43:01.957951+0000","last_undegraded":"2024-09-15T03:43:01.957951+0000","last_fullsized":"2024-09-15T03:43:01.957951+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:01.957828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:43:01.957828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:10.420536+0000","scrub_duration":0.0016180420000000001,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:43:00.057271+0000","last_change":"2024-09-15T03:42:20.126436+0000","last_active":"2024-09-15T03:43:00.057271+0000","last_peered":"2024-09-15T03:43:00.057271+0000","last_clean":"2024-09-15T03:43:00.057271+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:43:00.057271+0000","last_undegraded":"2024-09-15T03:43:00.057271+0000","last_fullsized":"2024-09-15T03:43:00.057271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:42:05.071899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:42:05.071899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:17.090770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"146'2","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-15T03:43:02.167075+0000","last_change":"2024-09-15T03:43:02.167048+0000","last_active":"2024-09-15T03:43:02.167075+0000","last_peered":"2024-09-15T03:43:02.167075+0000","last_clean":"2024-09-15T03:43:02.167075+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:43:02.167075+0000","last_undegraded":"2024-09-15T03:43:02.167075+0000","last_fullsized":"2024-09-15T03:43:02.167075+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'2","last_scrub_stamp":"2024-09-15T03:43:02.166968+0000","last_deep_scrub":"146'2","last_deep_scrub_stamp":"2024-09-15T03:43:02.166968+0000","last_clean_scrub_stamp":"2024-09-15T03:43:02.166968+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:19.638339+0000","scrub_duration":0.0029732690000000002,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":272,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-15T03:42:59.368961+0000","last_change":"2024-09-15T03:41:51.788478+0000","last_active":"2024-09-15T03:42:59.368961+0000","last_peered":"2024-09-15T03:42:59.368961+0000","last_clean":"2024-09-15T03:42:59.368961+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:42:59.368961+0000","last_undegraded":"2024-09-15T03:42:59.368961+0000","last_fullsized":"2024-09-15T03:42:59.368961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:41:51.788386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:41:51.788386+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021062139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304220,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067620,"statfs":{"total":96636764160,"available":96325242880,"internally_reserved":0,"allocated":2768896,"data_stored":1846506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.627},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.69699999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.42099999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.65000000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.71299999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.57499999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.436},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":0.55000000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.72399999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":28.465,"5min":28.465,"15min":28.465},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":667.26900000000001,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.503}]}]},{"osd":14,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303700,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068140,"statfs":{"total":96636764160,"available":96325775360,"internally_reserved":0,"allocated":2236416,"data_stored":1317578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.38300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.47599999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.65200000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:43:01 2024","interfaces":[{"interface":"back","average":{"1min":58.142000000000003,"5min":12.138,"15min":4.4699999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":979.04700000000003,"5min":979.04700000000003,"15min":979.04700000000003},"last":979.04700000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.44600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.433},{"interface":"front","average":{"1min":5.0359999999999996,"5min":5.0359999999999996,"15min":5.0359999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":97.417000000000002,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.48299999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.627}]},{"osd":12,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.45800000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.64000000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.67600000000000005}]}]},{"osd":13,"up_from":22,"seq":94489280543,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304212,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067628,"statfs":{"total":96636764160,"available":96325251072,"internally_reserved":0,"allocated":2760704,"data_stored":1845314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.66600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.70699999999999996},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.59499999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.434},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.501}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.54800000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.48499999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.752}]},{"osd":11,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.80700000000000005},{"interface":"front","average":{"1min":47.228999999999999,"5min":47.228999999999999,"15min":47.228999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":974.15300000000002,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.64100000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.58399999999999996},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.52600000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.68899999999999995}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.81799999999999995}]}]},{"osd":0,"up_from":21,"seq":90194313249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303700,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068140,"statfs":{"total":96636764160,"available":96325775360,"internally_reserved":0,"allocated":2232320,"data_stored":1317000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.747}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.439},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.60699999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.53200000000000003}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.76000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.54100000000000004,"15min":0.55300000000000005},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79700000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.622}]},{"osd":13,"last update":"Sun Sep 15 03:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65300000000000002,"15min":0.65100000000000002},"min":{"1min":0.47499999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.67300000000000004,"15min":0.66100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.86899999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.77800000000000002}]}]},{"osd":6,"up_from":21,"seq":90194313247,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303684,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068156,"statfs":{"total":96636764160,"available":96325791744,"internally_reserved":0,"allocated":2220032,"data_stored":1308062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.438},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.65800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.54600000000000004}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.377},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.41799999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.629},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55800000000000005}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.49099999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":54.079000000000001,"5min":54.079000000000001,"15min":54.079000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":909.03399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.50600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67500000000000004}]},{"osd":14,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39300000000000002},{"interface":"front","average":{"1min":41.031999999999996,"5min":41.031999999999996,"15min":41.031999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":686.33799999999997,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.57299999999999995}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.70099999999999996}]}]},{"osd":15,"up_from":21,"seq":90194313249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303680,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068160,"statfs":{"total":96636764160,"available":96325795840,"internally_reserved":0,"allocated":2215936,"data_stored":1307484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.38},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.434}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.878}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.95599999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70499999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.91600000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.69499999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.83799999999999997}]}]},{"osd":3,"up_from":68,"seq":292057776149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299604,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072236,"statfs":{"total":96636764160,"available":96329969664,"internally_reserved":0,"allocated":2236416,"data_stored":1316970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.75},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.84999999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.95499999999999996}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.879}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.67300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.89900000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.752},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.73699999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303684,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068156,"statfs":{"total":96636764160,"available":96325791744,"internally_reserved":0,"allocated":2220032,"data_stored":1308062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69899999999999995}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.50600000000000001}]},{"osd":9,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.876}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.61399999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.46600000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48799999999999999}]},{"osd":13,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.85799999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.84099999999999997}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.80700000000000005}]}]},{"osd":11,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303684,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068156,"statfs":{"total":96636764160,"available":96325791744,"internally_reserved":0,"allocated":2220032,"data_stored":1308062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.627},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1629999999999998,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.36799999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.39500000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":47.381,"5min":47.381,"15min":47.381},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":981.20500000000004,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.76700000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66700000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.47899999999999998}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.71299999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.79300000000000004}]},{"osd":15,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.57399999999999995}]}]},{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299524,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072316,"statfs":{"total":96636764160,"available":96330051584,"internally_reserved":0,"allocated":2220032,"data_stored":1307503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.48599999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.69299999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.38300000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.50800000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.67600000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.56599999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.63},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.58499999999999996}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.68500000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.65800000000000003}]}]},{"osd":1,"up_from":22,"seq":94489280543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303696,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068144,"statfs":{"total":96636764160,"available":96325779456,"internally_reserved":0,"allocated":2232320,"data_stored":1317000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.59799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.38800000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.754}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.315},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.35799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.67700000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.57599999999999996}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.41899999999999998},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.40699999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.55500000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.72099999999999997}]}]},{"osd":4,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303700,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068140,"statfs":{"total":96636764160,"available":96325775360,"internally_reserved":0,"allocated":2236416,"data_stored":1317578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.622},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63800000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.498},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.44700000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.60399999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.41199999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.65600000000000003}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.47299999999999998}]},{"osd":14,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":22.876000000000001,"5min":22.876000000000001,"15min":22.876000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":353.327,"5min":353.327,"15min":353.327},"last":0.69499999999999995},{"interface":"front","average":{"1min":61.741999999999997,"5min":61.741999999999997,"15min":61.741999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":974.78599999999994,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.73299999999999998}]},{"osd":15,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.58599999999999997}]}]},{"osd":5,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303696,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068144,"statfs":{"total":96636764160,"available":96325779456,"internally_reserved":0,"allocated":2232320,"data_stored":1317000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.51800000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.70099999999999996}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.41199999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.65400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.499},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48099999999999998}]},{"osd":13,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.58499999999999996}]},{"osd":14,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.68500000000000005}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.67300000000000004}]}]},{"osd":8,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303700,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068140,"statfs":{"total":96636764160,"available":96325775360,"internally_reserved":0,"allocated":2236416,"data_stored":1317578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.55500000000000005},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":0.498}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64400000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.81200000000000006}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":0.53500000000000003}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":0.69399999999999995}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.59499999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.48099999999999998}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.626},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.70899999999999996}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.79400000000000004}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303700,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068140,"statfs":{"total":96636764160,"available":96325775360,"internally_reserved":0,"allocated":2236416,"data_stored":1317578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.36199999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.67600000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.502}]},{"osd":7,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.61499999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.47799999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.46400000000000002}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":0.63},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":0.76700000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":0.747},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.56299999999999994}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":0.72699999999999998}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.71299999999999997},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":0.77800000000000002}]}]},{"osd":10,"up_from":22,"seq":94489280544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303700,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068140,"statfs":{"total":96636764160,"available":96325775360,"internally_reserved":0,"allocated":2236416,"data_stored":1317578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":10.291,"5min":10.291,"15min":10.291},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":176.59299999999999,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54400000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.371}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40000000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68799999999999994}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.629},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58599999999999997}]},{"osd":12,"last update":"Sun Sep 15 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.504}]},{"osd":13,"last update":"Sun Sep 15 03:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.54000000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91000000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":41.664000000000001,"15min":48.490000000000002},"min":{"1min":0.48499999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.91500000000000004,"5min":923.202,"15min":923.202},"last":640.80700000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:43:07.182 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T03:42:05.044126+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=5, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=58, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:43:07.182 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T03:42:05.088015+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=5, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=58, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:43:07.182 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T03:42:07.145656+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=58, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:43:07.182 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T03:41:55.835634+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=41, tm_sec=55, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=58, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:43:07.182 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T03:42:05.071899+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=5, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=58, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:43:07.182 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T03:41:51.788386+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=41, tm_sec=51, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=42, tm_sec=58, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:43:07.182 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:43:27.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:43:27.480 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-15T03:43:27.481 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-15T03:43:27.492 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-15T03:43:26.047159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":4857632,"kb_used_data":37464,"kb_used_omap":26,"kb_used_meta":4819941,"kb_avail":1505091808,"statfs":{"total":1546188226560,"available":1541214011392,"internally_reserved":0,"allocated":38363136,"data_stored":23028937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26928,"internal_metadata":4935620304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001811"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":153,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-15T03:43:18.057258+0000","last_change":"2024-09-15T03:43:07.065151+0000","last_active":"2024-09-15T03:43:18.057258+0000","last_peered":"2024-09-15T03:43:18.057258+0000","last_clean":"2024-09-15T03:43:18.057258+0000","last_became_active":"2024-09-15T03:40:46.659089+0000","last_became_peered":"2024-09-15T03:40:46.659089+0000","last_unstale":"2024-09-15T03:43:18.057258+0000","last_undegraded":"2024-09-15T03:43:18.057258+0000","last_fullsized":"2024-09-15T03:43:18.057258+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:07.065068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:43:07.065068+0000","last_clean_scrub_stamp":"2024-09-15T03:43:07.065068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:15.107601+0000","scrub_duration":0.001719117,"objects_trimmed":0,"snaptrim_duration":0.00029703900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:43:21.536749+0000","last_change":"2024-09-15T03:43:03.473350+0000","last_active":"2024-09-15T03:43:21.536749+0000","last_peered":"2024-09-15T03:43:21.536749+0000","last_clean":"2024-09-15T03:43:21.536749+0000","last_became_active":"2024-09-15T03:40:46.657366+0000","last_became_peered":"2024-09-15T03:40:46.657366+0000","last_unstale":"2024-09-15T03:43:21.536749+0000","last_undegraded":"2024-09-15T03:43:21.536749+0000","last_fullsized":"2024-09-15T03:43:21.536749+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:03.473234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:43:03.473234+0000","last_clean_scrub_stamp":"2024-09-15T03:43:03.473234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:12.454856+0000","scrub_duration":0.001735994,"objects_trimmed":0,"snaptrim_duration":0.00032889300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":119,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-15T03:43:17.369815+0000","last_change":"2024-09-15T03:43:03.445583+0000","last_active":"2024-09-15T03:43:17.369815+0000","last_peered":"2024-09-15T03:43:17.369815+0000","last_clean":"2024-09-15T03:43:17.369815+0000","last_became_active":"2024-09-15T03:42:41.277214+0000","last_became_peered":"2024-09-15T03:42:41.277214+0000","last_unstale":"2024-09-15T03:43:17.369815+0000","last_undegraded":"2024-09-15T03:43:17.369815+0000","last_fullsized":"2024-09-15T03:43:17.369815+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:03.445442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:43:03.445442+0000","last_clean_scrub_stamp":"2024-09-15T03:43:03.445442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:21.489679+0000","scrub_duration":0.001431229,"objects_trimmed":0,"snaptrim_duration":0.00032111100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-15T03:43:17.460148+0000","last_change":"2024-09-15T03:43:00.339427+0000","last_active":"2024-09-15T03:43:17.460148+0000","last_peered":"2024-09-15T03:43:17.460148+0000","last_clean":"2024-09-15T03:43:17.460148+0000","last_became_active":"2024-09-15T03:42:21.252159+0000","last_became_peered":"2024-09-15T03:42:21.252159+0000","last_unstale":"2024-09-15T03:43:17.460148+0000","last_undegraded":"2024-09-15T03:43:17.460148+0000","last_fullsized":"2024-09-15T03:43:17.460148+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:00.339350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:43:00.339350+0000","last_clean_scrub_stamp":"2024-09-15T03:43:00.339350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:22.051499+0000","scrub_duration":0.0012992660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"146'3","reported_seq":174,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:43:21.536794+0000","last_change":"2024-09-15T03:43:06.707753+0000","last_active":"2024-09-15T03:43:21.536794+0000","last_peered":"2024-09-15T03:43:21.536794+0000","last_clean":"2024-09-15T03:43:21.536794+0000","last_became_active":"2024-09-15T03:42:23.138547+0000","last_became_peered":"2024-09-15T03:42:23.138547+0000","last_unstale":"2024-09-15T03:43:21.536794+0000","last_undegraded":"2024-09-15T03:43:21.536794+0000","last_fullsized":"2024-09-15T03:43:21.536794+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'3","last_scrub_stamp":"2024-09-15T03:43:06.707645+0000","last_deep_scrub":"146'3","last_deep_scrub_stamp":"2024-09-15T03:43:06.707645+0000","last_clean_scrub_stamp":"2024-09-15T03:43:06.707645+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:19.463485+0000","scrub_duration":0.0030689490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:43:21.536718+0000","last_change":"2024-09-15T03:43:01.957922+0000","last_active":"2024-09-15T03:43:21.536718+0000","last_peered":"2024-09-15T03:43:21.536718+0000","last_clean":"2024-09-15T03:43:21.536718+0000","last_became_active":"2024-09-15T03:40:46.659062+0000","last_became_peered":"2024-09-15T03:40:46.659062+0000","last_unstale":"2024-09-15T03:43:21.536718+0000","last_undegraded":"2024-09-15T03:43:21.536718+0000","last_fullsized":"2024-09-15T03:43:21.536718+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:01.957828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:45.641258+0000","last_clean_scrub_stamp":"2024-09-15T03:43:01.957828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:10.420536+0000","scrub_duration":0.0016180420000000001,"objects_trimmed":0,"snaptrim_duration":0.00030229900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:43:21.536790+0000","last_change":"2024-09-15T03:43:05.668540+0000","last_active":"2024-09-15T03:43:21.536790+0000","last_peered":"2024-09-15T03:43:21.536790+0000","last_clean":"2024-09-15T03:43:21.536790+0000","last_became_active":"2024-09-15T03:42:20.126189+0000","last_became_peered":"2024-09-15T03:42:20.126189+0000","last_unstale":"2024-09-15T03:43:21.536790+0000","last_undegraded":"2024-09-15T03:43:21.536790+0000","last_fullsized":"2024-09-15T03:43:21.536790+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:43:05.668445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:43:05.668445+0000","last_clean_scrub_stamp":"2024-09-15T03:43:05.668445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:29.329194+0000","scrub_duration":0.001884646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"146'2","reported_seq":139,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:43:21.241281+0000","last_change":"2024-09-15T03:43:02.167048+0000","last_active":"2024-09-15T03:43:21.241281+0000","last_peered":"2024-09-15T03:43:21.241281+0000","last_clean":"2024-09-15T03:43:21.241281+0000","last_became_active":"2024-09-15T03:42:33.208990+0000","last_became_peered":"2024-09-15T03:42:33.208990+0000","last_unstale":"2024-09-15T03:43:21.241281+0000","last_undegraded":"2024-09-15T03:43:21.241281+0000","last_fullsized":"2024-09-15T03:43:21.241281+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'2","last_scrub_stamp":"2024-09-15T03:43:02.166968+0000","last_deep_scrub":"146'2","last_deep_scrub_stamp":"2024-09-15T03:43:02.166968+0000","last_clean_scrub_stamp":"2024-09-15T03:43:02.166968+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:19.638339+0000","scrub_duration":0.0029732690000000002,"objects_trimmed":0,"snaptrim_duration":0.00034895500000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"24'112","reported_seq":288,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-15T03:43:21.536694+0000","last_change":"2024-09-15T03:43:02.944714+0000","last_active":"2024-09-15T03:43:21.536694+0000","last_peered":"2024-09-15T03:43:21.536694+0000","last_clean":"2024-09-15T03:43:21.536694+0000","last_became_active":"2024-09-15T03:40:43.639255+0000","last_became_peered":"2024-09-15T03:40:43.639255+0000","last_unstale":"2024-09-15T03:43:21.536694+0000","last_undegraded":"2024-09-15T03:43:21.536694+0000","last_fullsized":"2024-09-15T03:43:21.536694+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'112","last_scrub_stamp":"2024-09-15T03:43:02.944621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:40:42.622204+0000","last_clean_scrub_stamp":"2024-09-15T03:43:02.944621+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:22.670228+0000","scrub_duration":0.0024408160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304300,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067540,"statfs":{"total":96636764160,"available":96325160960,"internally_reserved":0,"allocated":2850816,"data_stored":1894086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,7,8,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":7.024,"15min":8.1010000000000009},"min":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84699999999999998,"5min":170.74299999999999,"15min":170.74299999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.048,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.59199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":32.921999999999997,"5min":32.921999999999997,"15min":32.921999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":501.35199999999998,"5min":501.35199999999998,"15min":501.35199999999998},"last":0.93300000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.72999999999999998,"15min":0.75900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72899999999999998,"5min":4.6849999999999996,"15min":4.6849999999999996},"last":0.748},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.748,"15min":0.78000000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.65000000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.61399999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.36499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.69999999999999996,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":2.383,"15min":2.383},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":1.224,"15min":1.224},"last":0.80900000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":18.920000000000002,"15min":21.954999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.996,"5min":478.76900000000001,"15min":478.76900000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69199999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":1.421,"15min":1.421},"last":0.72999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.38500000000000001},{"interface":"front","average":{"1min":20.285,"5min":20.285,"15min":20.285},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":451.81099999999998,"5min":451.81099999999998,"15min":451.81099999999998},"last":0.76400000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.90300000000000002,"5min":1.458,"15min":1.458},"last":0.88400000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:43:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":22.908999999999999,"15min":26.613},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.90700000000000003,"5min":667.26900000000001,"15min":667.26900000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.67700000000000005}]}]},{"osd":14,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303796,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068044,"statfs":{"total":96636764160,"available":96325677056,"internally_reserved":0,"allocated":2334720,"data_stored":1374674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,4,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":2.923,"5min":2.923,"15min":2.923},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":45.768999999999998,"5min":45.768999999999998,"15min":45.768999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":61.835000000000001,"5min":61.835000000000001,"15min":61.835000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":25.007999999999999,"5min":5.4359999999999999,"15min":2.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":391.11799999999999,"5min":391.11799999999999,"15min":391.11799999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84799999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.55500000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68000000000000005,"5min":0.752,"15min":0.752},"last":0.748},{"interface":"front","average":{"1min":0.57999999999999996,"5min":39.015999999999998,"15min":45.421999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86299999999999999,"5min":913.03200000000004,"15min":913.03200000000004},"last":0.46899999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56000000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.625}]},{"osd":7,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.343,"5min":26.343,"15min":26.343},"last":0.73199999999999998}]},{"osd":8,"last update":"Sun Sep 15 03:43:01 2024","interfaces":[{"interface":"back","average":{"1min":58.142000000000003,"5min":12.138,"15min":4.4699999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":979.04700000000003,"5min":979.04700000000003,"15min":979.04700000000003},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.81000000000000005}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.61799999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.075,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":4.1459999999999999,"15min":4.7389999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.38300000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.77000000000000002}]},{"osd":12,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.73799999999999999,"15min":0.754},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91600000000000004,"5min":1.125,"15min":1.125},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.72099999999999997,"15min":0.72899999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82799999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.79600000000000004}]},{"osd":13,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":24.303000000000001,"15min":28.227},"min":{"1min":0.53300000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.181,"5min":560.85299999999995,"15min":560.85299999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.754,"15min":0.749},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.58999999999999997}]},{"osd":15,"last update":"Sun Sep 15 03:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0609999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.64000000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304308,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067532,"statfs":{"total":96636764160,"available":96325152768,"internally_reserved":0,"allocated":2859008,"data_stored":1902410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":7.5629999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":131.56899999999999,"5min":131.56899999999999,"15min":131.56899999999999},"last":0.621}]},{"osd":1,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.73399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77600000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.752,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.317,"15min":0.317},"max":{"1min":1.0169999999999999,"5min":1.143,"15min":1.143},"last":0.63500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":25.745999999999999,"5min":25.745999999999999,"15min":25.745999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":525.46400000000006,"5min":525.46400000000006,"15min":525.46400000000006},"last":0.59699999999999998},{"interface":"front","average":{"1min":12.858000000000001,"5min":12.858000000000001,"15min":12.858000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":255.047,"5min":255.047,"15min":255.047},"last":0.499}]},{"osd":4,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.097,"5min":1.234,"15min":1.234},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.1000000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.874}]},{"osd":5,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99299999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.98899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.54700000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.91300000000000003},{"interface":"front","average":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":9.5030000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":183.66800000000001,"5min":183.66800000000001,"15min":183.66800000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.0169999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.052}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.46100000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":14.593999999999999,"15min":16.882999999999999},"min":{"1min":0.51600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1699999999999999,"5min":379.39100000000002,"15min":379.39100000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.93100000000000005,"5min":33.034999999999997,"15min":38.386000000000003},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2090000000000001,"5min":886.51800000000003,"15min":886.51800000000003},"last":0.76600000000000001}]},{"osd":11,"last update":"Sun Sep 15 03:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.89800000000000002,"15min":0.89700000000000002},"min":{"1min":0.64800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2350000000000001,"5min":1.413,"15min":1.413},"last":0.73499999999999999},{"interface":"front","average":{"1min":11.672000000000001,"5min":40.116999999999997,"15min":44.857999999999997},"min":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":250.143,"5min":974.15300000000002,"15min":974.15300000000002},"last":0.83199999999999996}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84899999999999998,"15min":0.84299999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.2070000000000001,"5min":1.534,"15min":1.534},"last":0.89000000000000001},{"interface":"front","average":{"1min":20.734000000000002,"5min":4.8460000000000001,"15min":2.198},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":437.46899999999999,"5min":437.46899999999999,"15min":437.46899999999999},"last":0.80100000000000005}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.872,"15min":0.85499999999999998},"min":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2250000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.93100000000000005,"5min":10.835000000000001,"15min":12.484999999999999},"min":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.246,"5min":274.65100000000001,"15min":274.65100000000001},"last":0.86099999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.89400000000000002,"15min":0.88500000000000001},"min":{"1min":0.63300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3109999999999999,"5min":1.611,"15min":1.611},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.012,"5min":0.90100000000000002,"15min":0.88300000000000001},"min":{"1min":0.72299999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.321,"5min":1.454,"15min":1.454},"last":0.90500000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":2347008,"data_stored":1383612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,6,7,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":33.689,"15min":39.207000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.93999999999999995,"5min":703.98400000000004,"15min":703.98400000000004},"last":0.48999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":27.812999999999999,"5min":27.812999999999999,"15min":27.812999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":344.27499999999998,"5min":344.27499999999998,"15min":344.27499999999998},"last":0.80200000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":6,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":47.802999999999997,"5min":10.044,"15min":3.7509999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":945.53499999999997,"5min":945.53499999999997,"15min":945.53499999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.65600000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":20.260999999999999,"15min":23.535},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86799999999999999,"5min":419.01499999999999,"15min":419.01499999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":45.804000000000002,"5min":9.6519999999999992,"15min":3.6259999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":905.48299999999995,"5min":905.48299999999995,"15min":905.48299999999995},"last":0.63900000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:45 2024","interfaces":[{"interface":"back","average":{"1min":44.587000000000003,"5min":44.587000000000003,"15min":44.587000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":748.04600000000005,"5min":748.04600000000005,"15min":748.04600000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.28299999999999997}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66300000000000003,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90700000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81899999999999995,"5min":1.327,"15min":1.327},"last":0.44700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.54100000000000004,"15min":0.55300000000000005},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79700000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.41799999999999998}]},{"osd":13,"last update":"Sun Sep 15 03:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65300000000000002,"15min":0.65100000000000002},"min":{"1min":0.47499999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.67300000000000004,"15min":0.66100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61699999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:43:15 2024","interfaces":[{"interface":"back","average":{"1min":49.192,"5min":49.192,"15min":49.192},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":565.15999999999997,"5min":565.15999999999997,"15min":565.15999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.84199999999999997}]},{"osd":15,"last update":"Sun Sep 15 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":3.3159999999999998,"15min":3.7530000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88900000000000001,"5min":62.659999999999997,"15min":62.659999999999997},"last":0.754},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":297.51299999999998}]}]},{"osd":6,"up_from":21,"seq":90194313251,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303796,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068044,"statfs":{"total":96636764160,"available":96325677056,"internally_reserved":0,"allocated":2334720,"data_stored":1374674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[4,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":29.693000000000001,"5min":29.693000000000001,"15min":29.693000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":495.29399999999998,"5min":495.29399999999998,"15min":495.29399999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.498}]},{"osd":4,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":30.251000000000001,"15min":35.195999999999998},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":668.923,"15min":668.923},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.753,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.69599999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":12.904,"15min":14.952},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0049999999999999,"5min":290.32400000000001,"15min":290.32400000000001},"last":0.83499999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":17.472999999999999,"5min":3.9830000000000001,"15min":1.7350000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":254.161,"5min":254.161,"15min":254.161},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.74199999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.66700000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:42:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79700000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.53600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73099999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.77700000000000002}]},{"osd":10,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":43.058,"5min":43.058,"15min":43.058},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":762.63599999999997,"5min":762.63599999999997,"15min":762.63599999999997},"last":0.53900000000000003}]},{"osd":11,"last update":"Sun Sep 15 03:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.70899999999999996},{"interface":"front","average":{"1min":12.055999999999999,"5min":45.673999999999999,"15min":51.277000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":194.81399999999999,"5min":909.03399999999999,"15min":909.03399999999999},"last":0.67800000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":13,"last update":"Sun Sep 15 03:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.75700000000000001}]},{"osd":14,"last update":"Sun Sep 15 03:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.86499999999999999,"5min":0.876,"15min":0.876},"last":0.51100000000000001},{"interface":"front","average":{"1min":12.513999999999999,"5min":35.328000000000003,"15min":39.130000000000003},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":202.37100000000001,"5min":686.33799999999997,"15min":686.33799999999997},"last":0.58499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.91600000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.90900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.79500000000000004}]}]},{"osd":15,"up_from":21,"seq":90194313253,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303776,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068064,"statfs":{"total":96636764160,"available":96325697536,"internally_reserved":0,"allocated":2314240,"data_stored":1364580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,5,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.754,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":36.487000000000002,"15min":42.478000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.751,"5min":764.62199999999996,"15min":764.62199999999996},"last":0.35299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.79200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:41:48 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82999999999999996,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.70399999999999996}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.436},{"interface":"front","average":{"1min":0.64200000000000002,"5min":15.004,"15min":17.398},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92600000000000005,"5min":305.18200000000002,"15min":305.18200000000002},"last":0.51400000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":47.200000000000003,"15min":54.988},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85599999999999998,"5min":782.05200000000002,"15min":782.05200000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46500000000000002}]},{"osd":8,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":59.442,"5min":59.442,"15min":59.442},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":762.18499999999995,"5min":762.18499999999995,"15min":762.18499999999995},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60599999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.84599999999999997}]},{"osd":10,"last update":"Sun Sep 15 03:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.857,"5min":1.857,"15min":1.857},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":21.629999999999999},"last":0.54000000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89400000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.499},{"interface":"front","average":{"1min":12.439,"5min":3.0289999999999999,"15min":1.4610000000000001},"min":{"1min":0.44700000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":235.608,"5min":235.608,"15min":235.608},"last":0.52700000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.67100000000000004},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":59.548999999999999,"5min":12.470000000000001,"15min":4.6239999999999997},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":981.43100000000004,"5min":981.43100000000004,"15min":981.43100000000004},"last":0.73999999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:53 2024","interfaces":[{"interface":"back","average":{"1min":7.0979999999999999,"5min":1.974,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":129.565,"5min":129.565,"15min":129.565},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99299999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.61399999999999999}]}]},{"osd":3,"up_from":68,"seq":292057776153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299716,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":297406,"kb_avail":94072124,"statfs":{"total":96636764160,"available":96329854976,"internally_reserved":0,"allocated":2351104,"data_stored":1383582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1368,"internal_metadata":304544424},"hb_peers":[0,1,2,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.372},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.84599999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":39.892000000000003,"5min":39.892000000000003,"15min":39.892000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":501.02300000000002,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.753}]},{"osd":5,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68200000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.89700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":10,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.81100000000000005}]},{"osd":11,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.79300000000000004}]},{"osd":12,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.78600000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.88300000000000001}]}]},{"osd":7,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303780,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068060,"statfs":{"total":96636764160,"available":96325693440,"internally_reserved":0,"allocated":2318336,"data_stored":1365158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.79800000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":23.553999999999998,"5min":23.553999999999998,"15min":23.553999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":436.91899999999998,"5min":436.91899999999998,"15min":436.91899999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.96299999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.94299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.42399999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.48699999999999999}]},{"osd":8,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.98199999999999998,"5min":1.046,"15min":1.046},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.234,"15min":1.234},"last":0.53100000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.63200000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.90100000000000002,"5min":1.03,"15min":1.03},"last":0.59999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.35799999999999998,"5min":0.312,"15min":0.312},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.36899999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.119,"5min":1.22,"15min":1.22},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0229999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.44}]},{"osd":13,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.78300000000000003,"15min":0.78000000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.752},"min":{"1min":0.61399999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1779999999999999,"5min":1.25,"15min":1.25},"last":0.69399999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.55900000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":32.170999999999999,"15min":37.395000000000003},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2410000000000001,"5min":740.69799999999998,"15min":740.69799999999998},"last":0.58499999999999996}]},{"osd":15,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.501},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.145,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.64200000000000002}]}]},{"osd":11,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303780,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068060,"statfs":{"total":96636764160,"available":96325693440,"internally_reserved":0,"allocated":2318336,"data_stored":1365158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.496}]},{"osd":1,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":57.634,"5min":12.006,"15min":4.4009999999999998},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":892.99000000000001,"5min":892.99000000000001,"15min":892.99000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82599999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.502}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.39600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.98299999999999998,"5min":3.177,"15min":3.177},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":4,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":29.814,"5min":29.814,"15min":29.814},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":501.01600000000002,"5min":501.01600000000002,"15min":501.01600000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80400000000000005}]},{"osd":6,"last update":"Sun Sep 15 03:42:04 2024","interfaces":[{"interface":"back","average":{"1min":42.923999999999999,"5min":42.923999999999999,"15min":42.923999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":973.46600000000001,"5min":973.46600000000001,"15min":973.46600000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.2360000000000002,"5min":3.2360000000000002,"15min":3.2360000000000002},"last":0.88}]},{"osd":7,"last update":"Sun Sep 15 03:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73999999999999999,"5min":3.1629999999999998,"15min":3.1629999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.37}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":26.846,"15min":31.215},"min":{"1min":0.45600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90200000000000002,"5min":557.40099999999995,"15min":557.40099999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.105,"15min":0.92800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":30.241,"5min":30.241,"15min":30.241},"last":0.69099999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":38.042999999999999,"15min":44.268000000000001},"min":{"1min":0.51700000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.93899999999999995,"5min":981.20500000000004,"15min":981.20500000000004},"last":0.626},{"interface":"front","average":{"1min":12.007999999999999,"5min":3.0619999999999998,"15min":1.571},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":227.989,"5min":227.989,"15min":227.989},"last":0.63900000000000001}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.66300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.70999999999999996}]},{"osd":12,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":33.451999999999998,"15min":38.920000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":694.31399999999996,"15min":694.31399999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.746}]},{"osd":13,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.367,"15min":1.478},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":11.574,"15min":11.574},"last":0.48799999999999999}]},{"osd":14,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.747,"15min":0.74199999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.79600000000000004}]},{"osd":15,"last update":"Sun Sep 15 03:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.73999999999999999,"15min":0.746},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.46800000000000003}]}]},{"osd":12,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303780,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068060,"statfs":{"total":96636764160,"available":96325693440,"internally_reserved":0,"allocated":2318336,"data_stored":1364599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44},{"interface":"front","average":{"1min":36.750999999999998,"5min":36.750999999999998,"15min":36.750999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":760.27800000000002,"5min":760.27800000000002,"15min":760.27800000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55800000000000005,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78900000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.55400000000000005},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.49199999999999999}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":7.7569999999999997,"5min":2.0600000000000001,"15min":1.1100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":152.04900000000001,"5min":152.04900000000001,"15min":152.04900000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":1.139,"15min":1.139},"last":0.69699999999999995}]},{"osd":7,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":980.91399999999999},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":9,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":12.643000000000001,"5min":3.0939999999999999,"15min":1.502},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":254.131,"5min":254.131,"15min":254.131},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.65400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.437}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.80000000000000004}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":8.8659999999999997,"15min":10.231999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0529999999999999,"5min":201.73599999999999,"15min":201.73599999999999},"last":0.56499999999999995}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.266,"15min":1.3740000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91400000000000003,"5min":14.640000000000001,"15min":14.640000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0589999999999999,"15min":1.123},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.96299999999999997,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.64400000000000002}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":33.442,"15min":38.904000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0900000000000001,"5min":818.68700000000001,"15min":818.68700000000001},"last":0.82399999999999995}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.579999999999998,"5min":30.669,"15min":33.017000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":334.38499999999999,"5min":703.52499999999998,"15min":703.52499999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.006,"5min":1.286,"15min":1.286},"last":0.628}]}]},{"osd":1,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303792,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068048,"statfs":{"total":96636764160,"available":96325681152,"internally_reserved":0,"allocated":2330624,"data_stored":1374096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":496.51999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.90800000000000003,"5min":1.069,"15min":1.069},"last":0.54000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.39300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86099999999999999,"5min":1.27,"15min":1.27},"last":0.5},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.38200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.434}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":31.41,"5min":31.41,"15min":31.41},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":676.83699999999999,"5min":676.83699999999999,"15min":676.83699999999999},"last":0.74399999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":21.585000000000001,"5min":4.8410000000000002,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":399.51799999999997,"5min":399.51799999999997,"15min":399.51799999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85699999999999998,"5min":1.105,"15min":1.105},"last":0.81100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94899999999999995,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":960.52300000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":8.8300000000000001,"15min":10.19},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.96599999999999997,"5min":214.91,"15min":214.91},"last":0.46800000000000003}]},{"osd":7,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.41899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":1.306,"15min":1.306},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.1060000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68899999999999995,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":1.155,"15min":1.155},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.70899999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96599999999999997,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.79400000000000004}]},{"osd":9,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97099999999999997,"5min":1.516,"15min":1.516},"last":0.61799999999999999},{"interface":"front","average":{"1min":35.646000000000001,"5min":7.7060000000000004,"15min":3.0489999999999999},"min":{"1min":0.35899999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":665.41499999999996,"5min":665.41499999999996,"15min":665.41499999999996},"last":0.72199999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.752,"15min":0.76200000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.998,"5min":1.746,"15min":1.746},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.81299999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.006,"5min":1.54,"15min":1.54},"last":0.71299999999999997}]},{"osd":11,"last update":"Sun Sep 15 03:42:25 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":2.085,"15min":2.085},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":30.010000000000002,"5min":30.010000000000002,"15min":30.010000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.85099999999999998}]}]},{"osd":4,"up_from":22,"seq":94489280548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303796,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068044,"statfs":{"total":96636764160,"available":96325677056,"internally_reserved":0,"allocated":2334720,"data_stored":1374674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.42299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":39.113,"5min":39.113,"15min":39.113},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":576.76199999999994,"5min":576.76199999999994,"15min":576.76199999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.60499999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76200000000000001,"5min":1.119,"15min":1.119},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.871,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.45300000000000001}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.71099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91500000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":4.319,"15min":4.9409999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":74.475999999999999,"15min":74.475999999999999},"last":0.71799999999999997}]},{"osd":7,"last update":"Sun Sep 15 03:41:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.76400000000000001}]},{"osd":8,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":54.087000000000003,"5min":54.087000000000003,"15min":54.087000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":801.00300000000004,"5min":801.00300000000004,"15min":801.00300000000004},"last":0.47499999999999998}]},{"osd":9,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59699999999999998}]},{"osd":10,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.63800000000000001},{"interface":"front","average":{"1min":31.684000000000001,"5min":31.684000000000001,"15min":31.684000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":463.887,"5min":463.887,"15min":463.887},"last":0.73799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.81699999999999995,"15min":0.83999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95999999999999996,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.53300000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0229999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.73599999999999999}]},{"osd":12,"last update":"Sun Sep 15 03:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.016}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":25.574999999999999,"5min":5.819,"15min":2.5259999999999998},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":448.57499999999999,"5min":448.57499999999999,"15min":448.57499999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":52.496000000000002,"5min":11.228999999999999,"15min":4.351},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":932.00199999999995,"5min":932.00199999999995,"15min":932.00199999999995},"last":0.56799999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":18.439,"15min":21.396999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.028,"5min":353.327,"15min":353.327},"last":0.749},{"interface":"front","average":{"1min":0.76200000000000001,"5min":49.545999999999999,"15min":57.676000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2490000000000001,"5min":974.78599999999994,"15min":974.78599999999994},"last":0.61699999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.86899999999999999,"15min":0.88100000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.878,"15min":0.89600000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.77500000000000002}]}]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303792,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068048,"statfs":{"total":96636764160,"available":96325681152,"internally_reserved":0,"allocated":2330624,"data_stored":1374096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.439},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:42:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58299999999999996,"15min":0.57299999999999995},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86299999999999999,"5min":1.046,"15min":1.046},"last":0.67300000000000004}]},{"osd":6,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.42499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.63200000000000001}]},{"osd":7,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.96599999999999997}]},{"osd":8,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.49199999999999999}]},{"osd":9,"last update":"Sun Sep 15 03:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:43:15 2024","interfaces":[{"interface":"back","average":{"1min":19.529,"5min":4.5309999999999997,"15min":2.0310000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":357.73599999999999,"5min":357.73599999999999,"15min":357.73599999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":44.789999999999999,"5min":9.5649999999999995,"15min":3.694},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":836.80999999999995,"5min":836.80999999999995,"15min":836.80999999999995},"last":0.66200000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":12,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.61499999999999999}]},{"osd":13,"last update":"Sun Sep 15 03:43:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1789999999999998,"5min":1.0309999999999999,"15min":0.83999999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":27.888999999999999,"5min":27.888999999999999,"15min":27.888999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.75600000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0249999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57299999999999995}]},{"osd":14,"last update":"Sun Sep 15 03:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.80300000000000005,"15min":0.79000000000000004},"min":{"1min":0.629,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.57599999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.78600000000000003}]},{"osd":15,"last update":"Sun Sep 15 03:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.75900000000000001},"min":{"1min":0.66700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.755},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.66300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.77000000000000002}]}]},{"osd":8,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303816,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068024,"statfs":{"total":96636764160,"available":96325656576,"internally_reserved":0,"allocated":2351104,"data_stored":1384190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56599999999999995},{"interface":"front","average":{"1min":27.158999999999999,"5min":5.9379999999999997,"15min":2.4009999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":531.13,"5min":531.13,"15min":531.13},"last":0.501}]},{"osd":2,"last update":"Sun Sep 15 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":39.040999999999997,"5min":39.040999999999997,"15min":39.040999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":500.89999999999998,"5min":500.89999999999998,"15min":500.89999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2089999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":50.808999999999997,"5min":50.808999999999997,"15min":50.808999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":903.59500000000003,"5min":903.59500000000003,"15min":903.59500000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.40699999999999997}]},{"osd":6,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84699999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.73499999999999999}]},{"osd":7,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":18.353999999999999,"15min":21.309999999999999},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":398.71699999999998,"15min":398.71699999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.46300000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52900000000000003}]},{"osd":9,"last update":"Sun Sep 15 03:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":41.003,"15min":47.728999999999999},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81999999999999995,"5min":908.44200000000001,"15min":908.44200000000001},"last":0.42699999999999999}]},{"osd":10,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":78.944000000000003,"5min":78.944000000000003,"15min":78.944000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":14.851000000000001,"5min":14.851000000000001,"15min":14.851000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":256.41000000000003,"5min":256.41000000000003,"15min":256.41000000000003},"last":0.59999999999999998}]},{"osd":11,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":0.876,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":7.3029999999999999,"15min":8.4019999999999992},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86199999999999999,"5min":149.09,"15min":149.09},"last":0.48199999999999998}]},{"osd":12,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46100000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67300000000000004,"15min":0.66000000000000003},"min":{"1min":0.52100000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94599999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.92300000000000004,"5min":1.024,"15min":1.024},"last":0.66000000000000003}]},{"osd":14,"last update":"Sun Sep 15 03:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.68999999999999995,"15min":0.68700000000000006},"min":{"1min":0.52300000000000002,"5min":0.495,"15min":0.495},"max":{"1min":0.88900000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":51.404000000000003,"5min":10.848000000000001,"15min":4.0890000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":887.44500000000005,"5min":887.44500000000005,"15min":887.44500000000005},"last":0.69199999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":9,"up_from":22,"seq":94489280549,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303796,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068044,"statfs":{"total":96636764160,"available":96325677056,"internally_reserved":0,"allocated":2334720,"data_stored":1374674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58699999999999997,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":34.143999999999998,"5min":34.143999999999998,"15min":34.143999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":572.42399999999998,"5min":572.42399999999998,"15min":572.42399999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":29.646000000000001,"5min":29.646000000000001,"15min":29.646000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":496.08100000000002,"5min":496.08100000000002,"15min":496.08100000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.67300000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.746},{"interface":"front","average":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":40.167000000000002,"5min":40.167000000000002,"15min":40.167000000000002},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":8.7780000000000005,"15min":10.127000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84999999999999998,"5min":151.80600000000001,"15min":151.80600000000001},"last":0.60299999999999998}]},{"osd":7,"last update":"Sun Sep 15 03:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.74299999999999999,"15min":0.74399999999999999},"min":{"1min":0.498,"5min":0.438,"15min":0.438},"max":{"1min":0.97899999999999998,"5min":1.014,"15min":1.014},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.74099999999999999,"15min":0.747},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90900000000000003,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.57999999999999996}]},{"osd":8,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.745},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92800000000000005,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0209999999999999,"5min":1.907,"15min":1.907},"last":0.5}]},{"osd":10,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78000000000000003,"15min":0.78500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.002,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.748,"5min":6.0110000000000001,"15min":6.8879999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.048,"5min":98.858000000000004,"15min":98.858000000000004},"last":0.68700000000000006}]},{"osd":11,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":18.529,"5min":6.2649999999999997,"15min":4.2210000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":337.03300000000002,"5min":337.03300000000002,"15min":337.03300000000002},"last":0.62},{"interface":"front","average":{"1min":0.81699999999999995,"5min":11.618,"15min":13.417999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0660000000000001,"5min":203.91399999999999,"15min":203.91399999999999},"last":0.70399999999999996}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":13,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":16.349,"5min":3.9350000000000001,"15min":1.8660000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":297.36500000000001,"5min":297.36500000000001,"15min":297.36500000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":16.361000000000001,"5min":3.9049999999999998,"15min":1.829},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":297.28899999999999,"5min":297.28899999999999,"15min":297.28899999999999},"last":0.84099999999999997}]},{"osd":14,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.879},"min":{"1min":0.46999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.175,"5min":1.258,"15min":1.258},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":4.7859999999999996,"15min":5.4459999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.105,"5min":73.846999999999994,"15min":73.846999999999994},"last":0.75600000000000001}]},{"osd":15,"last update":"Sun Sep 15 03:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.58899999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1000000000000001,"5min":1.208,"15min":1.208},"last":0.80400000000000005},{"interface":"front","average":{"1min":20.795999999999999,"5min":4.8479999999999999,"15min":2.1899999999999999},"min":{"1min":0.66900000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":379.226,"5min":379.226,"15min":379.226},"last":0.76600000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303796,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068044,"statfs":{"total":96636764160,"available":96325677056,"internally_reserved":0,"allocated":2334720,"data_stored":1374674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,3,4,5,6,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":8.3599999999999994,"15min":9.6470000000000002},"min":{"1min":0.46200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81899999999999995,"5min":176.59299999999999,"15min":176.59299999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.52500000000000002,"15min":0.503},"min":{"1min":0.42699999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.48999999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.76300000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.621},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70299999999999996}]},{"osd":5,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.70299999999999996,"15min":0.71599999999999997},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":2.391,"15min":2.391},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.70699999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Sep 15 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.39700000000000002,"15min":0.38300000000000001},"min":{"1min":0.313,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.28699999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.33100000000000002}]},{"osd":9,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82699999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.42799999999999999}]},{"osd":11,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.629}]},{"osd":12,"last update":"Sun Sep 15 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":55.119,"5min":11.487,"15min":4.2149999999999999},"min":{"1min":0.44700000000000001,"5min":0.377,"15min":0.377},"max":{"1min":818.97000000000003,"5min":818.97000000000003,"15min":818.97000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78600000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.64600000000000002}]},{"osd":13,"last update":"Sun Sep 15 03:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.54000000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91000000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":41.664000000000001,"15min":48.490000000000002},"min":{"1min":0.48499999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.91500000000000004,"5min":923.202,"15min":923.202},"last":0.56699999999999995}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":15,"last update":"Sun Sep 15 03:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":13.407999999999999,"15min":15.534000000000001},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94099999999999995,"5min":287.86900000000003,"15min":287.86900000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.60999999999999999},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:43:27.498 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T03:43:27.790 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T03:43:27.790 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T03:43:27.790 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T03:43:27.790 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:27.826 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T03:43:27.826 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T03:43:27.827 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:27.899 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T03:43:27.899 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-15T03:43:27.899 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:27.967 INFO:tasks.ceph.osd.8:Stopped 2024-09-15T03:43:27.968 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-15T03:43:27.968 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.046 INFO:tasks.ceph.osd.12:Stopped 2024-09-15T03:43:28.046 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T03:43:28.046 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.082 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T03:43:28.082 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T03:43:28.082 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.160 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T03:43:28.160 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-15T03:43:28.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.231 INFO:tasks.ceph.osd.9:Stopped 2024-09-15T03:43:28.232 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-15T03:43:28.232 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.309 INFO:tasks.ceph.osd.13:Stopped 2024-09-15T03:43:28.309 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T03:43:28.309 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.342 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T03:43:28.343 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-15T03:43:28.343 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.418 INFO:tasks.ceph.osd.6:Stopped 2024-09-15T03:43:28.418 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-15T03:43:28.418 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.485 INFO:tasks.ceph.osd.10:Stopped 2024-09-15T03:43:28.486 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-15T03:43:28.486 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.555 INFO:tasks.ceph.osd.14:Stopped 2024-09-15T03:43:28.555 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:43:28.555 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.592 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:43:28.593 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-15T03:43:28.593 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.662 INFO:tasks.ceph.osd.7:Stopped 2024-09-15T03:43:28.662 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-15T03:43:28.662 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.742 INFO:tasks.ceph.osd.11:Stopped 2024-09-15T03:43:28.742 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-15T03:43:28.742 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.811 INFO:tasks.ceph.osd.15:Stopped 2024-09-15T03:43:28.811 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T03:43:28.812 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-15T03:43:28.812 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.837 INFO:tasks.ceph.mgr.y:Stopped 2024-09-15T03:43:28.837 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T03:43:28.837 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.880 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T03:43:28.880 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T03:43:28.881 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:43:28.881 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.894 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:43:28.894 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:43:28.895 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.909 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:43:28.909 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:43:28.909 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:43:28.923 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:43:28.923 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T03:43:28.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-09-15T03:43:28.972 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:43:28.972 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T03:43:29.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:43:29.194 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T03:43:29.278 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:43:29.278 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-15T03:43:29.344 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:43:29.344 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-15T03:43:29.402 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:43:29.402 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T03:43:29.545 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:43:29.545 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T03:43:29.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:43:29.609 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-15T03:43:29.666 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:43:29.667 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-15T03:43:29.724 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:43:29.724 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T03:43:29.857 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:43:29.858 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-15T03:43:29.946 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:43:29.947 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-15T03:43:30.004 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:43:30.004 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-15T03:43:30.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:43:30.087 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T03:43:30.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:43:30.354 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-15T03:43:30.459 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:43:30.459 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-15T03:43:30.518 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:43:30.518 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-15T03:43:30.569 INFO:tasks.ceph:Archiving mon data... 2024-09-15T03:43:30.570 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/data/mon.a.tgz 2024-09-15T03:43:30.571 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-15T03:43:30.590 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.qSqftl95xd 2024-09-15T03:43:30.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qSqftl95xd 2024-09-15T03:43:30.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.qSqftl95xd 2024-09-15T03:43:30.849 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.qSqftl95xd is 438KB 2024-09-15T03:43:30.946 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.qSqftl95xd 2024-09-15T03:43:30.962 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/data/mon.b.tgz 2024-09-15T03:43:30.962 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-15T03:43:30.981 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.JN5v4Zj593 2024-09-15T03:43:30.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.JN5v4Zj593 2024-09-15T03:43:31.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.JN5v4Zj593 2024-09-15T03:43:31.307 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.JN5v4Zj593 is 620KB 2024-09-15T03:43:31.406 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.JN5v4Zj593 2024-09-15T03:43:31.422 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/data/mon.c.tgz 2024-09-15T03:43:31.422 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-15T03:43:31.440 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.prSqwHXmCp 2024-09-15T03:43:31.441 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.prSqwHXmCp 2024-09-15T03:43:31.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.prSqwHXmCp 2024-09-15T03:43:31.742 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.prSqwHXmCp is 620KB 2024-09-15T03:43:31.845 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.prSqwHXmCp 2024-09-15T03:43:31.860 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T03:43:31.860 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:43:31.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:43:31.865 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:43:31.902 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:43:31.931 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T03:43:31.932 DEBUG:teuthology.orchestra.run.smithi005:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:43:31.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:43:31.937 DEBUG:teuthology.orchestra.run.smithi078:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:43:31.964 INFO:teuthology.orchestra.run.smithi064.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:43:31.965 INFO:teuthology.orchestra.run.smithi005.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:43:31.967 DEBUG:teuthology.orchestra.run.smithi105:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:43:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:43:32.003 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:43:32.009 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T03:43:32.012 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi005/crash 2024-09-15T03:43:32.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:43:32.041 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi064/crash 2024-09-15T03:43:32.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:43:32.068 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi078/crash 2024-09-15T03:43:32.069 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:43:32.094 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi105/crash 2024-09-15T03:43:32.095 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:43:32.240 INFO:tasks.ceph:Compressing logs... 2024-09-15T03:43:32.241 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:43:32.244 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:43:32.246 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:43:32.249 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:43:32.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-15T03:43:32.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:43:32.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T03:43:32.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T03:43:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-15T03:43:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-15T03:43:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:43:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.log: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:43:32.275 INFO:teuthology.orchestra.run.smithi078.stderr: 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:43:32.277 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T03:43:32.277 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-15T03:43:32.277 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-15T03:43:32.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:43:32.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T03:43:32.277 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:43:32.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T03:43:32.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:43:32.278 INFO:teuthology.orchestra.run.smithi064.stderr: 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:43:32.293 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32855.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32495.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32455.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32855.log.gz 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32695.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32495.log.gz 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32455.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32455.log.gz 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32695.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39509.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32695.log.gz 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36496.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/objectstore_tool.39509.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.43699.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.0.36496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32364.log 2024-09-15T03:43:32.294 INFO:teuthology.orchestra.run.smithi105.stderr: 75.8% -- replaced with /var/log/ceph/ceph-client.0.36496.log.gz 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/bluestore_tool.43699.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.37083.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32364.log.gz 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33014.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/objectstore_tool.37083.log: /var/log/ceph/ceph-client.admin.33014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33014.log.gz 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32735.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.43227.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.15.log: /var/log/ceph/ceph-client.admin.32735.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32735.log.gz 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32775.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/bluestore_tool.43227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32535.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32775.log.gz 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32324.log 2024-09-15T03:43:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32535.log.gz 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32414.log 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/bluestore_tool.43699.log.gz 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32324.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32324.log.gz 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/objectstore_tool.37985.log 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32414.log.gz 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.42759.log 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/objectstore_tool.37985.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/bluestore_tool.42759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32655.log 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr: 45.3% -- replaced with /var/log/ceph/bluestore_tool.42759.log.gz 2024-09-15T03:43:32.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.11.log: 93.5% -- replaced with /var/log/ceph/bluestore_tool.43227.log.gz 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr: 93.2% -- replaced with /var/log/ceph/objectstore_tool.39509.log.gz 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.38747.log 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32655.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32655.log.gz 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32974.log 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/objectstore_tool.38747.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32974.log.gz 2024-09-15T03:43:32.310 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32815.log 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32615.log 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32815.log: /var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32815.log.gz 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32575.log 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32615.log: 26.1% -- replaced with /var/log/ceph/ceph-client.admin.32615.log.gz 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.42779.log 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.32575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32575.log.gz 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/bluestore_tool.42779.log: 93.3% -- replaced with /var/log/ceph/objectstore_tool.37985.log.gz 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/objectstore_tool..log: 93.5% -- replaced with /var/log/ceph/objectstore_tool.37083.log.gz 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr: 93.8% -- replaced with /var/log/ceph/bluestore_tool.42779.log.gz 2024-09-15T03:43:32.311 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.3.log: 93.4% -- replaced with /var/log/ceph/objectstore_tool.38747.log.gz 2024-09-15T03:43:32.355 INFO:teuthology.orchestra.run.smithi105.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-09-15T03:43:32.368 INFO:teuthology.orchestra.run.smithi078.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T03:43:32.374 INFO:teuthology.orchestra.run.smithi105.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T03:43:32.390 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T03:43:32.390 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48438.log 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50582.log 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49952.log 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48438.log.gz 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38466.log 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50582.log.gz 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50834.log 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49952.log.gz 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41333.log 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38466.log.gz 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50834.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50246.log -- replaced with /var/log/ceph/ceph-client.admin.50834.log.gz 2024-09-15T03:43:32.391 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39829.log 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.41333.log.gz 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49020.log 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50246.log.gz 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46417.log 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.39829.log.gz 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49020.log.gz 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50666.log 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37095.log 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr: 22.2% -- replaced with /var/log/ceph/ceph-client.admin.46417.log.gz 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47390.log 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50666.log.gz 2024-09-15T03:43:32.392 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48984.log 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37095.log.gz 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47390.log.gz 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41292.log 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48446.log 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48984.log.gz 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49700.log 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41292.log.gz 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48446.log.gz 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44161.log 2024-09-15T03:43:32.393 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49700.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49059.log 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48890.log 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44161.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49059.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44703.log 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48890.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44703.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37084.log 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-09-15T03:43:32.394 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45664.log 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37731.log 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37084.log.gz 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45425.log 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45664.log.gz 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46718.log 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37731.log.gz 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45425.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.43024.log -- replaced with /var/log/ceph/ceph-client.admin.45425.log.gz 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37759.log 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46718.log.gz 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr: 22.1% -- replaced with /var/log/ceph/ceph-client.admin.43024.log.gz 2024-09-15T03:43:32.395 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50288.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37759.log.gz 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49658.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 23.2% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50792.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50288.log.gz 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50960.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49658.log.gz 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46146.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50792.log.gz 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50960.log.gz 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46629.log 2024-09-15T03:43:32.396 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46146.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38301.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.44661.log 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47642.log 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46629.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40852.log 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44661.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46041.log 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47642.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40852.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37750.log 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46041.log.gz 2024-09-15T03:43:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45313.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37750.log.gz 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46345.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48437.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51254.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46345.log.gz 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37079.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48437.log.gz 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46844.log 2024-09-15T03:43:32.398 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51254.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49106.log 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37079.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51336.log 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46844.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48428.log 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49106.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51336.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47936.log 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.48428.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50330.log 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-09-15T03:43:32.399 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47936.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47936.log.gz 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50330.log.gz 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46970.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46928.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45383.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46970.log.gz 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46928.log.gz 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46021.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45383.log.gz 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47768.log 2024-09-15T03:43:32.400 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37721.log 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46021.log.gz 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47768.log.gz 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39630.log 0.0% 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37721.log.gz 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40341.log: /var/log/ceph/ceph-client.admin.49069.log 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: 22.2% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39630.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46387.log -- replaced with /var/log/ceph/ceph-client.admin.39630.log.gz 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49510.log 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49069.log.gz 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50456.log 2024-09-15T03:43:32.401 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43434.log 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49510.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47852.log 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50456.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47600.log 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43434.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39022.log 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47852.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33263.log 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47600.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47726.log 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39022.log.gz 2024-09-15T03:43:32.402 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.33263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40172.log 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33263.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50498.log 0.0% 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47726.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43496.log 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40172.log: 22.9% -- replaced with /var/log/ceph/ceph-client.admin.40172.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38180.log 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50498.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr: 22.9% -- replaced with /var/log/ceph/ceph-client.admin.43496.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44785.log 0.0% 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38180.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-09-15T03:43:32.403 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50078.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44785.log.gz 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48410.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47138.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50078.log.gz 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38495.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48410.log.gz 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47138.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47138.log.gz 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38728.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38495.log.gz 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2024-09-15T03:43:32.404 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38728.log.gz 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44783.log 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51044.log 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46187.log: 42.0% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49075.log 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr: 22.5% -- replaced with /var/log/ceph/ceph-client.admin.44783.log.gz 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47558.log 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51044.log.gz 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49075.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49075.log.gz 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47558.log.gzgzip 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47810.log 2024-09-15T03:43:32.405 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42734.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.47810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47810.log.gz 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36619.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47054.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42734.log: 22.9% -- replaced with /var/log/ceph/ceph-client.admin.42734.log.gz 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47348.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36619.log.gz 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47054.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47054.log.gz -5 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39788.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37696.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47348.log.gz 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38455.log 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39788.log.gz 2024-09-15T03:43:32.406 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37050.log 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37696.log.gz 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37515.log 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38455.log.gz 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45312.log 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37050.log.gz 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37515.log.gz 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37071.log 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49014.log 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45312.log.gz 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48395.log 2024-09-15T03:43:32.407 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37071.log.gz 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49014.log.gz 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45042.log 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33143.log 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48395.log.gz 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45042.log.gz 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37733.log 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.33143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37081.log 2024-09-15T03:43:32.408 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33143.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37733.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44784.log 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.37081.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47684.log 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37081.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45560.log 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44784.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47684.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46886.log 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45805.log 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45560.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46886.log.gz 2024-09-15T03:43:32.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49046.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr: 22.5% -- replaced with /var/log/ceph/ceph-client.admin.45805.log.gz 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50204.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50120.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49046.log.gz 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50204.log.gz 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44323.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45225.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50120.log.gz 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48422.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44323.log.gz 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47474.log 2024-09-15T03:43:32.410 INFO:teuthology.orchestra.run.smithi005.stderr: 22.9% -- replaced with /var/log/ceph/ceph-client.admin.45225.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39691.log 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48422.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39126.log 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47474.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46565.log 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr: 22.6% -- replaced with /var/log/ceph/ceph-client.admin.39691.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43277.log 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39126.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49868.log 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46565.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43277.log.gz 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38938.log 2024-09-15T03:43:32.411 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48104.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40748.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38938.log.gz 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48104.log.gz 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38768.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47222.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38768.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48062.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38768.log.gz 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47222.log.gz 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43946.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47264.log 2024-09-15T03:43:32.412 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48062.log.gz 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39247.log 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43946.log.gz 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47264.log.gz 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50750.log 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37090.log 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39247.log: 22.5% -- replaced with /var/log/ceph/ceph-client.admin.39247.log.gz 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50750.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26984.log 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50750.log.gz 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37090.log.gz 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45445.log 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49080.log 2024-09-15T03:43:32.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.tmp-client.admin.26984.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26984.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39248.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr: 22.2% -- replaced with /var/log/ceph/ceph-client.admin.45445.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49080.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39248.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49418.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr: 21.9% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32865.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36670.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49418.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.32865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50624.log 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32865.log.gz 2024-09-15T03:43:32.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45540.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36670.log.gz 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48971.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50624.log.gz 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45540.log.gz 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36754.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48971.log.gz 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36712.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37772.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36754.log.gz 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-09-15T03:43:32.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36712.log.gz 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46802.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37772.log.gz 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37087.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48433.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46802.log.gz 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51128.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37680.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48433.log.gz 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47096.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51128.log.gz 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50918.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37680.log: /var/log/ceph/ceph-client.admin.47096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43318.log 2024-09-15T03:43:32.416 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47096.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37680.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51170.log 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50918.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43318.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51170.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51170.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47516.log 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47516.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47516.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48020.log 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49826.log 2024-09-15T03:43:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48020.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48020.log.gz -5 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51296.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49826.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49826.log.gz 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.42922.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51296.log.gz 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44520.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42922.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42922.log.gz 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42452.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44520.log.gz 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48416.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37687.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42452.log.gz 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50036.log 2024-09-15T03:43:32.418 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48416.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37687.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49100.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50036.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49100.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46760.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44458.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46760.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46760.log.gz 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43988.log 2024-09-15T03:43:32.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44458.log.gz 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41954.log 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43988.log: gzip -5 --verbose -- 22.8% -- replaced with /var/log/ceph/ceph-client.admin.43988.log.gz 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph.log 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41954.log: gzip 22.4% -- replaced with /var/log/ceph/ceph-client.admin.41954.log.gz 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47432.log 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48424.log 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46503.log 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47432.log.gz 2024-09-15T03:43:32.420 INFO:teuthology.orchestra.run.smithi005.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38302.log 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48424.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45931.log 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46503.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42807.log 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38302.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45931.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49078.log 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42807.log.gz 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51086.log 2024-09-15T03:43:32.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44586.log 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49078.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37060.log 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51086.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48413.log 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr: 22.1% -- replaced with /var/log/ceph/ceph-client.admin.44586.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37086.log 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37060.log: /var/log/ceph/ceph-client.admin.48413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48403.log 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48413.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37060.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37086.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37086.log.gz 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-09-15T03:43:32.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48403.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42267.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39651.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51002.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42267.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.42267.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50162.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51002.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50162.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50162.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50372.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50372.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50372.log.gz 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38447.log 2024-09-15T03:43:32.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38447.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.41688.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38447.log.gz 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45184.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49068.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41688.log.gz 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49742.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45184.log.gz 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49068.log.gz 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46262.log 2024-09-15T03:43:32.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49742.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38220.log 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46262.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46262.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51212.log 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38220.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47306.log 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51212.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.49994.log -- replaced with /var/log/ceph/ceph-client.admin.51212.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47306.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49784.log 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49994.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49994.log.gz 2024-09-15T03:43:32.425 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47012.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49784.log.gz 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37741.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47012.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47012.log.gz 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37741.log.gz 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39064.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40921.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44519.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39064.log.gz 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44519.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44519.log.gz 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43872.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49910.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43729.log 2024-09-15T03:43:32.426 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48425.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr: 22.5% -- replaced with /var/log/ceph/ceph-client.admin.43872.log.gz 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46282.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43729.log.gz 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50708.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48425.log: /var/log/ceph/ceph-client.admin.46282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46545.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48425.log.gz 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46282.log.gz 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50708.log.gz 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37055.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46671.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46545.log: /var/log/ceph/ceph-client.admin.37055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38896.log 2024-09-15T03:43:32.427 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46545.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37055.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46671.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38240.log 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48397.log 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38896.log: /var/log/ceph/ceph-client.admin.38240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38453.log 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38896.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38240.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49616.log 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48397.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47894.log 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38453.log: /var/log/ceph/ceph-client.admin.49616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36570.log 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38453.log.gz 2024-09-15T03:43:32.428 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49616.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47894.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37053.log 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47978.log 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36570.log: /var/log/ceph/ceph-client.admin.37053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50876.log 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36570.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37053.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47978.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37094.log 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44365.log 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50876.log: /var/log/ceph/ceph-client.admin.37094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39420.log 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50876.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37094.log.gz 2024-09-15T03:43:32.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44365.log: 22.2% -- replaced with /var/log/ceph/ceph-client.admin.44365.log.gz 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37739.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39420.log: /var/log/ceph/ceph-client.admin.42381.log: gzip -5 --verbose -- 22.8% -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.48436.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39420.log.gz 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37739.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43360.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48436.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.43831.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48436.log.gz 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50414.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr: 22.1% -- replaced with /var/log/ceph/ceph-client.admin.43360.log.gz 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37724.log 2024-09-15T03:43:32.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43831.log: /var/log/ceph/ceph-client.admin.50414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47180.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43831.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50414.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37724.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38476.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37637.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47180.log: /var/log/ceph/ceph-client.admin.38476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47180.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38460.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38476.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37637.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37096.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38460.log: /var/log/ceph/ceph-client.admin.37096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48802.log 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38460.log.gz 2024-09-15T03:43:32.431 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37096.log.gz 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48445.log 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr: 22.1% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48802.log: /var/log/ceph/ceph-client.admin.48445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37064.log 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48445.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48802.log.gz 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38140.log 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37064.log: /var/log/ceph/ceph-client.admin.46104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37064.log.gz 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50540.log 2024-09-15T03:43:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38140.log.gz 2024-09-15T03:43:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37617.log 2024-09-15T03:43:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50540.log.gz 2024-09-15T03:43:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49574.log 2024-09-15T03:43:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37617.log.gz 2024-09-15T03:43:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49574.log.gz 2024-09-15T03:43:32.433 INFO:teuthology.orchestra.run.smithi064.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T03:43:32.538 INFO:teuthology.orchestra.run.smithi105.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T03:43:32.581 INFO:teuthology.orchestra.run.smithi005.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T03:43:32.656 INFO:teuthology.orchestra.run.smithi078.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-15T03:43:32.670 INFO:teuthology.orchestra.run.smithi005.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-15T03:43:32.745 INFO:teuthology.orchestra.run.smithi105.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-15T03:43:32.751 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T03:43:32.759 INFO:teuthology.orchestra.run.smithi078.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T03:43:32.776 INFO:teuthology.orchestra.run.smithi064.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-15T03:43:32.805 INFO:teuthology.orchestra.run.smithi105.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-15T03:43:32.813 INFO:teuthology.orchestra.run.smithi005.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T03:43:32.817 INFO:teuthology.orchestra.run.smithi078.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-15T03:43:32.845 INFO:teuthology.orchestra.run.smithi078.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-15T03:43:32.847 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T03:43:32.847 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m0.586s 2024-09-15T03:43:32.847 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m2.029s 2024-09-15T03:43:32.847 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.082s 2024-09-15T03:43:32.856 INFO:teuthology.orchestra.run.smithi064.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T03:43:32.896 INFO:teuthology.orchestra.run.smithi005.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T03:43:32.905 INFO:teuthology.orchestra.run.smithi105.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-15T03:43:32.906 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-09-15T03:43:32.907 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m0.641s 2024-09-15T03:43:32.907 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m1.980s 2024-09-15T03:43:32.907 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.116s 2024-09-15T03:43:32.923 INFO:teuthology.orchestra.run.smithi064.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-15T03:43:32.924 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-15T03:43:32.925 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.664s 2024-09-15T03:43:32.925 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m2.287s 2024-09-15T03:43:32.925 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.086s 2024-09-15T03:43:33.046 INFO:teuthology.orchestra.run.smithi005.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-15T03:43:33.048 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-15T03:43:33.048 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m0.789s 2024-09-15T03:43:33.049 INFO:teuthology.orchestra.run.smithi005.stderr:user 0m2.492s 2024-09-15T03:43:33.049 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m0.239s 2024-09-15T03:43:33.049 INFO:tasks.ceph:Archiving logs... 2024-09-15T03:43:33.049 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi005/log 2024-09-15T03:43:33.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:43:33.692 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi064/log 2024-09-15T03:43:33.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:43:34.091 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi078/log 2024-09-15T03:43:34.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:43:34.463 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi105/log 2024-09-15T03:43:34.464 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:43:34.827 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:43:34.836 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:43:34.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:43:34.839 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:43:34.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:43:34.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:43:34.963 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:43:34.972 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:43:34.972 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:34.975 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:34.978 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:34.981 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:43:34.990 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:34.992 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:34.994 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:34.995 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-09-15T03:43:34.995 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +211us[ +225us] +/- 51ms 2024-09-15T03:43:34.995 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 -113us[ -113us] +/- 66ms 2024-09-15T03:43:34.995 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -153us[ -153us] +/- 76ms 2024-09-15T03:43:34.995 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:34.995 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:34.997 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:43:34.997 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:34.997 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-15T03:43:34.998 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +196us[ +199us] +/- 51ms 2024-09-15T03:43:34.998 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -116us[ -114us] +/- 66ms 2024-09-15T03:43:34.998 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -151us[ -149us] +/- 76ms 2024-09-15T03:43:34.998 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:35.000 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:35.000 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-09-15T03:43:35.000 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +208us[ +215us] +/- 51ms 2024-09-15T03:43:35.000 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -114us[ -114us] +/- 66ms 2024-09-15T03:43:35.000 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -150us[ -150us] +/- 76ms 2024-09-15T03:43:35.000 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:35.001 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:43:35.001 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-09-15T03:43:35.001 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +204us[ +210us] +/- 51ms 2024-09-15T03:43:35.001 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -121us[ -121us] +/- 66ms 2024-09-15T03:43:35.001 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -151us[ -144us] +/- 76ms 2024-09-15T03:43:35.001 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:43:35.003 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:43:35.012 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:43:35.012 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:43:35.021 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:43:35.116 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:43:35.189 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:43:35.261 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:43:35.334 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:43:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:35.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:43:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:35.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:43:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:35.435 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:43:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:35.464 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:43:35.475 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:43:35.484 INFO:teuthology.task.internal:Duration was 628.139842 seconds 2024-09-15T03:43:35.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:43:35.493 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:43:35.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:43:35.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:43:35.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:43:35.502 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:43:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:43:35.544 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:43:35.545 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:43:35.977 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:43:35.978 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-09-15T03:43:35.978 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:43:36.003 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-09-15T03:43:36.003 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:43:36.028 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T03:43:36.028 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:43:36.060 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-09-15T03:43:36.060 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:43:36.091 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:43:36.091 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:43:36.094 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:43:36.097 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:43:36.102 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:43:36.159 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:43:36.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:43:36.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:43:36.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:43:36.174 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:43:36.305 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:43:36.316 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:43:36.317 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:43:36.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:43:36.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:43:36.395 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:43:36.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:43:36.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.437 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.439 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.456 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-09-15T03:43:36.459 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-09-15T03:43:36.460 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-09-15T03:43:36.483 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-09-15T03:43:36.506 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:36.555 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:36.570 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:36.585 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:43:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:43:36.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:43:36.613 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:43:36.614 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi005 2024-09-15T03:43:36.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:43:36.649 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi064 2024-09-15T03:43:36.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:43:36.684 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi078 2024-09-15T03:43:36.684 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:43:36.717 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905296/remote/smithi105 2024-09-15T03:43:36.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:43:36.752 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:43:36.752 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:43:36.755 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:43:36.757 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:43:36.759 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:43:36.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:43:36.812 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:43:36.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:43:36.823 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:43:36.824 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:43:36.826 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:43:36.829 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:43:36.831 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:43:36.839 INFO:teuthology.orchestra.run.smithi005.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:43 /home/ubuntu/cephtest 2024-09-15T03:43:36.841 INFO:teuthology.orchestra.run.smithi064.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:43 /home/ubuntu/cephtest 2024-09-15T03:43:36.843 INFO:teuthology.orchestra.run.smithi078.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:43 /home/ubuntu/cephtest 2024-09-15T03:43:36.857 INFO:teuthology.orchestra.run.smithi105.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:43 /home/ubuntu/cephtest 2024-09-15T03:43:36.858 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:43:36.868 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:43:36.891 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 628.1398422718048 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:43:36.891 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:43:37.122 INFO:teuthology.run:pass